Note to Mashable: Online polls are garbage

Mashable offers some of the smartest online commentary and how-tos about social media on the Web.

Except for its online polls.

As statisticians know, most open-community online polls are garbage. You cannot generate a reliable sample for generalizing, and the results basically mean nothing.

Take Mashable’s latest declaration: “Nexus One crushes the iPhone 3GS in reader vote.” It posted an online poll in its Web Faceoff asking, “Who would win in a fight: Nexus One or iPhone 3GS?”

A few of the problems with this poll:

  • There is no control over the sample. Anyone with a computer and online access could click on the survey. For a poll to be statistically valid, the sample has to come from a population of users. It should not be self-selected; the researchers should randomly select people from the population to participate in the survey. As a result, this poll is subject to manipulation.
  • We don’t know who is answering the poll. Yes, 10,000 users responded to the poll. But who are they? Have they actually used both phones? Do they know the difference among all the models? And which sites referred the most users to this poll?
  • The question is vague. What does the poll question mean? Do users think Nexus One will outsell the iPhone? That it offers better apps? That it is easier to use? That it costs less? We don’t know because users could interpret “fight” a hundred different ways.

As Robert Niles has noted, open-community online polls are great for engaging your users and sparking conversation. But it is inappropriate and misleading to report the results as “news.”


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s