NBC News experiments with serious opinion polling through Facebook

As the 2012 elections finally begin, NBC News has been polling the public to see what issues concern them the most. What’s unusual is that it collected the data not by phone interviews, but through Facebook.

The polling is one piece of a larger partnership between NBC and Facebook, including a co-sponsored Republican candidates debate on “Meet The Press” this Sunday. MTP is crowdsourcing questions for the candidates through comments on its Facebook page, and the debate will stream live online along a Facebook live chat window.

Some early results of the New Hampshire polling appeared on “Meet The Press” last Sunday.

The attempt at statistically sound polling is the most noteworthy innovation so far. It raises the question: Is it possible to conduct a scientifically valid opinion poll through a social networking site?

So far, online polling has been a wasteland for “external validity” — the ability to say responses from the sample group accurately represent the views of the total population.

The simple daily polls embedded in the right-hand sidebars of many news sites do not even attempt external validity. Professional pollsters derisively nickname them SLOP (for “self-selected opinion polls”), because “respondents who volunteer to participate in such surveys tend to be more extreme or otherwise very different in their views than those who do not,” says the American Association for Public Opinion Research.

This is what the research poll looks like when presented to a Facebook user.

Facebook, however, may offer hope for a better type of online survey. For example, here’s how Ryan Osborn, senior director of digital media at NBC News, and Facebook explained their recent poll to me:

• Targeted sample. The poll is only offered to a preselected sample of users who are of voting age, with separate samples taken from residents of Iowa, New Hampshire and nationwide. Selected users see the poll in the right-hand sidebar of Facebook.
• Brand free. It is labeled only as a “research poll” with no NBC News branding, to avoid skewing responses based on a person’s perception of NBC or the fact that the results would air on TV.
• No horse race. The poll asks people to pick the campaign issue most important to them — a less contentious and complicated question than which candidate they support, Osborn said.
• Weighted results. The poll was circulated over three brief, separate periods over the course of a couple months. Results are cross-tabulated by age and gender, and weighted to account for any undersampled demographics in those categories.

In all, it seems like a well-thought-out process crafted by knowledgeable people in NBC’s polling division and Facebook’s measurement solutions team. That said, I don’t think this model is going to replace the decades-old NBC News/Wall Street Journal Poll anytime soon.

There are at least a few limitations to Facebook polling that make it less reliable than traditional polling.

One is that Facebook only knows what it knows. It’s great that Facebook knows the age and gender of each user. But it might not always know their ethnicity, or their political party affiliation. Those are attributes a pollster might ask about in a phone interview to properly screen or weight results.

It’s also limited by length — one survey question at a time right now — whereas a phone poll can ask many questions and cross-tabulate responses (“people who said X issue is most important were most likely to support Y candidate”). Professional pollsters also do little things, like rotating the order of multiple-choice answers or alternating how a question is phrased, to avoid biased results.

The biggest hurdle for this type of scientific research, though, is that Facebook is only Facebook. Yes, it’s huge, and people of all ages are on it. But it’s still just one closed social network that doesn’t reach the whole population.

“The real question is, are the 800 million people on Facebook representative of the general population? I think they’re getting pretty close, if not already there,” Osborn said. In the meantime, polling experts at NBC “have worked very hard to normalize it and really treat it as an editorially sound sample.”

Earlier: News organizations embrace social data in 2012 election coverage