NBC News experiments with serious opinion polling through Facebook

As the 2012 elections finally begin, NBC News has been polling the public to see what issues concern them the most. What’s unusual is that it collected the data not by phone interviews, but through Facebook.

The polling is one piece of a larger partnership between NBC and Facebook, including a co-sponsored Republican candidates debate on “Meet The Press” this Sunday. MTP is crowdsourcing questions for the candidates through comments on its Facebook page, and the debate will stream live online along a Facebook live chat window.

Some early results of the New Hampshire polling appeared on “Meet The Press” last Sunday.

The attempt at statistically sound polling is the most noteworthy innovation so far. It raises the question: Is it possible to conduct a scientifically valid opinion poll through a social networking site?

So far, online polling has been a wasteland for “external validity” — the ability to say responses from the sample group accurately represent the views of the total population.

The simple daily polls embedded in the right-hand sidebars of many news sites do not even attempt external validity. Professional pollsters derisively nickname them SLOP (for “self-selected opinion polls”), because “respondents who volunteer to participate in such surveys tend to be more extreme or otherwise very different in their views than those who do not,” says the American Association for Public Opinion Research.

This is what the research poll looks like when presented to a Facebook user.

Facebook, however, may offer hope for a better type of online survey. For example, here’s how Ryan Osborn, senior director of digital media at NBC News, and Facebook explained their recent poll to me:

  • Targeted sample. The poll is only offered to a preselected sample of users who are of voting age, with separate samples taken from residents of Iowa, New Hampshire and nationwide. Selected users see the poll in the right-hand sidebar of Facebook.
  • Brand free. It is labeled only as a “research poll” with no NBC News branding, to avoid skewing responses based on a person’s perception of NBC or the fact that the results would air on TV.
  • No horse race. The poll asks people to pick the campaign issue most important to them — a less contentious and complicated question than which candidate they support, Osborn said.
  • Weighted results. The poll was circulated over three brief, separate periods over the course of a couple months. Results are cross-tabulated by age and gender, and weighted to account for any undersampled demographics in those categories.

In all, it seems like a well-thought-out process crafted by knowledgeable people in NBC’s polling division and Facebook’s measurement solutions team. That said, I don’t think this model is going to replace the decades-old NBC News/Wall Street Journal Poll anytime soon.

There are at least a few limitations to Facebook polling that make it less reliable than traditional polling.

One is that Facebook only knows what it knows. It’s great that Facebook knows the age and gender of each user. But it might not always know their ethnicity, or their political party affiliation. Those are attributes a pollster might ask about in a phone interview to properly screen or weight results.

It’s also limited by length — one survey question at a time right now — whereas a phone poll can ask many questions and cross-tabulate responses (“people who said X issue is most important were most likely to support Y candidate”). Professional pollsters also do little things, like rotating the order of multiple-choice answers or alternating how a question is phrased, to avoid biased results.

The biggest hurdle for this type of scientific research, though, is that Facebook is only Facebook. Yes, it’s huge, and people of all ages are on it. But it’s still just one closed social network that doesn’t reach the whole population.

“The real question is, are the 800 million people on Facebook representative of the general population? I think they’re getting pretty close, if not already there,” Osborn said. In the meantime, polling experts at NBC “have worked very hard to normalize it and really treat it as an editorially sound sample.”

Earlier: News organizations embrace social data in 2012 election coverage

We have made it easy to comment on posts, however we require civility and encourage full names to that end (first initial, last name is OK). Please read our guidelines here before commenting.

  • Anonymous

    Many of the issues you mention are problems in traditional phone surveys as well.  Not everyone has a land-line phone.  People can and do lie to human pollsters.  And why couldn’t Facebook rotate answer order or ask variations of a question?

  • Anonymous

    The U.S. currently has a population of almost 313 million (see http://www.census.gov).  When talking about horserace or issue polls related to presidential elections, those living in the U.S. is what’s important.  And while it sounds impressive (but irresponsible, Mr. Osborn) to throw out a 800 million number, what matters is how many of them live in the U.S.  Even then you have to consider how those on Facebook, as a group, differ from those NOT on Facebook.  Add to that a whole assortment of other issues, you start to realize that Facebook is far from an adequate solution for polling the general population.  I do this for a living and while I wish it was. . . it is not.     

  • http://twitter.com/researchrants Research Rants

    I’d be concerned if they were trying to extrapolate Facebook results to apply to the whole population, but I don’t see anything wrong with using Facebook to fill in gaps (i.e., younger voters, cell phone only voters, etc) who are difficult or even impossible to reach via other methods.

  • Anonymous

    FB is so…2009. I deactivated last year, my younger teen did this fall. I tired of providing free content for M. Zuckerberg to sell ads alongside. I also grew distrustful of multiple “accidental” privacy snafus, which were accompanied by convoluted and increasingly disingenuous explanations. FB is the AOL of social media ( or the McDonald’s) – it’s a rush to enter a new world, at first. But the gateway vehicle {AOL for many, back in the day} is just that – a first foray. I am more into Twitter and Tumblr for now, and photo-sharing sites. NBC seems four years (or more) out of step by implying that “everyone” is on FB. 

  • Stormy Hatch

    Facebook doesn’t actually know what it knows, only what users tell it. I know that some users are not the age they claim to be, and presumably there are many other things on facebook profiles that are not..let’s say…….exactly accurate. It is fun to play with, however, and now that I’ve finally started using it, found is a great way to stay in touch with friends all over the globe. But it certainly has limitations, esp. concerning the validity of polling, among other things.


    NBC is right in polling Facebook posters. 800 million daily posters should have a fair good idea of our issues are for the average citizen. T. Luis