Television executives are increasingly concerned that paper diary measurement, a method from the 1950s to track TV viewing, excludes significant segments of the U.S. population, thus producing inaccurate ratings.
Diaries were never problem-free. Participants are asked to write in once a day what they watch on TV and when. The issues include inaccurate self-reporting, absence of households that should be represented in the samples, households declining to answer and long wait times for the data. The effort to analyze the diaries is massive: Nielsen uses diary measurement in 154 markets and processes two million diaries a year during “sweeps” periods.
Council for Research Excellence (CRE), a group of media measurement researchers including Nielsen executives, highlighted the biggest problem to date affecting diary data: a shortage of Blacks, Hispanics and young people among the participants.
How effective are diary measurements?
CRE addressed the issue at a “mini-summit” last week Tuesday in New York City. Based on 11 years of Nielsen data collected from 31 diary markets, CRE showed random errors have steadily increased.
Over time, local broadcast and cable ratings adopted a 10 percent margin of error that has become the acceptable standard used in advertising negotiations.
One of CRE’s research studies found only 11.3 percent of diary entries in a day fall within the accepted 10 percent error margin. For primetime viewing hours, the accuracy for entries is somewhat better, with 26 percent within the acceptable range. For evening and late night newscasts, accuracy fell within the acceptable margin at 18.1 percent and 20.7 percent, respectively.
The numbers, however, are not reassuring.
“Local TV diary, on a household basis, measures about 30 percent of all viewing in the U.S.,” Rick Ducey, managing director of research analysis for BIA/Kelsey, said at last week’s summit. “About 12 percent of total TV ad spending is based on diaries, so it’s a pretty significant economic force.”
Diaries aren’t yet obsolete because they provide a good enough estimation of TV viewing habits. More accurate measurement devices like people meters and “code readers” are on the horizon. These devices send information automatically to Nielsen about the shows people are actually watching based on the signals their televisions emit.
“A lot of people both on the buying side and the selling side get very frustrated at trying to use the diary,” said Billy McDowell, vice president of research at Raycom Media and chair of CRE Local measurement committee. Much of the frustration comes from the degree of inconsistency in the numbers.
However, rolling out the new devices has been more challenging than Nielsen expected, said Matt O’Grady, executive vice president and managing director of local media for the ratings service.
Non-respondents are different
Nielsen saw a rise in the accuracy of its ratings data after changing its sampling strategy from one based on telephone landlines to home addresses because more households have eschewed landlines in favor of mobile phones.
However, it isn’t easy to fix non-response bias, which introduces errors when people who are chosen to fill out diaries don’t participate or don’t answer all questions.
Nielsen uses demographic and market data to extrapolate the diary-based ratings to the greater U.S. population. This works as long as the households sampled represent a cross-section of the country.
But what happens if diaries overlook portions of the population? Census statistics may be of no help if the people excluded are fundamentally different in their viewing habits. And that’s what CRE found.
CRE embarked on a two-year, $2.1 million study with RTI International to answer a key question: Among people selected to write in diaries, what are the differences between people who participate and those who don’t?
Researchers looked at “responders” (or “Intabs”), versus “non-responders” (or “NonIntabs”) in three markets: Dallas, Albuquerque and Paducah.
Diary responders tend to:
• Include more whites than non-responders
• Be over 50 years old
• Be better educated
• Have no children at home
• Own their homes for more than 10 years
• Have a landline and one cell phone
• Watch networks like NBC, FOX, ABC and CBS
• Have cable TV at home
Non-responders are inclined to:
• Have larger household sizes with children at home
• Be renters
• Be younger
• Include higher numbers of Blacks and Hispanics
• Lack a landline phone but have at least one cell phone at home
• Have several electronic devices like computers, MP3 players and gaming consoles
• Watch networks like Univision, BET, MTV, Cartoon Network and HBO and less CBS and NBC than responders
• Have high-speed Internet at home and watch TV shows on the Internet
• Watch TV in groups, often at a friend’s house, or in restaurants and bars
How the different groups affect the numbers
Non-responders, who include the fastest growing population groups in the U.S., watch less TV than responders, whether during early viewing hours, prime time or overall. Further, the diaries don’t reflect homes that don’t have TV sets because these are excluded from the sample.
All of these factors increase the likelihood of ratings errors and raise the suspicion that the diaries are resulting in inaccurate ratings.
Concerns about the diaries’ reliability do not bode well for TV stations dependent on ratings-based advertising while challenged by Internet competition and aging audiences.
However, Nielsen is looking into which households are excluded, how to recruit them and “how to take the diary and evolve it,” said Nielsen’s chief methodologist Michael Link.
“The real way you’re going to correct non-response bias is upfront by finding better ways to recruit households at the beginning, not doing statistical adjustments at the back end,” he said.
The industry will continue to use diary data as long as local TV stations and advertisers “make their living based on diary information,” McDowell said, even while doubts persist.
Writer’s note: The link to the study presentation has been updated with slides from the CRE summit.
Comments