Why it’s time to stop romanticizing & begin measuring investigative journalism’s impact

Charles Lewis, one of the luminaries of nonprofit investigative journalism, sees a culture clash brewing as the sector continues to grow, covering what shrinking legacy media may miss and, more recently, innovating with powerful reporting techniques.

On the one hand, foundations big and small want metrics that demonstrate results analogous to assessments they apply to arts projects, social service initiatives and advocacy work.

On the other hand, Lewis wrote in a white paper last month, “veteran editors and reporters, particularly of the investigative ilk, have an inherent, almost visceral dislike of audience measurement and engagement strategies.” Instead they see themselves “as intrepid hunter-gatherers of information” who overcome a host of obstacles to produce important, even heroic, journalism.

The conflict might be academic were it not for the current state of play in nonprofit funding. Established nonprofit news sites need second and third rounds of support from foundations, and startups find foundations “feeling a bit overwhelmed and besieged by proliferating prospective grantees,” Lewis wrote.

In an e-mail interview, Lewis added, “Subjectivity is a serious occupational hazard for any grantor attempting to measure impact…Some foundations seem to be somewhat obsessed with these questions and issues, and others, not so much (especially smaller foundations with very few staff)”  But he expects the level of scrutiny to keep rising.

The white paper, which Lewis co-wrote with Hilary Niles, assesses the problems of measuring impact journalism and proposes the starting outlines of best practices. Written for the Investigative Reporting Workshop at American University, and underwritten by the McCormick Foundation, the paper is a lengthy but worthwhile read.  Caroline O’Donovan did an excellent job summarizing it in this Nieman Lab piece.

I’m not going to try to cover the entire scope, but here are a few themes:

Difficulties: If impact and outcomes are the true markers of an effective investigation, quantitative indicators of quality are likely to miss the mark. “Targeted reach” to decision makers may be more important than broad readership numbers. Sometimes remedial action on the problem spotlighted takes years rather than months. The investigative reporting process is more time-consuming and open-ended than advocacy work, which often assumes a desired outcome and marshals evidence to make the case.

Proxies: Audience is at least a starting point, with the qualifier that fluff may outdraw substance. Lewis and Niles recommend engagement as a more sophisticated measure of reach. Mentions, links and conversations are particularly susceptible to measurement in the digital space — and can indicate that the work in question didn’t fall with a thud in an empty forest.

Models: The authors point to several pilot projects with titles like “The Center for What Works” and “Spreading the Zing” that tackle the challenge of measuring social outcomes. Perhaps, Lewis and Niles argue, some of the framework and classification of desirable outcomes can be adapted by individual projects trying to make their case rather than reinvented from scratch.

Best practices: Lewis and Niles suggest that the Wisconsin Center for Investigative Journalism is well-along in documenting the “ripple effect” of its best work. As explained by the center’s Lauren Hasler in a 10-minute video, this consists of investing time and a little money into capturing data on the spread of a story and creating a narrative with evidence of what the investigation accomplished.

The authors give passing reference to what I think is a potential ace in the hole for the nonprofits: crowd-sourcing and crowd-funding. Though they yield input rather than outcome measures, both are direct evidence of an engaged audience, willing to contribute money for investigative work or do some of it for free. Crowd-sourcing and crowd-funding fit the digital era and the non-commercial idealism of nonprofit news organizations.

Also, while Lewis and Niles are focused on the nonprofit sector, I am curious whether the assessment challenge should be applied to investigative efforts at newspapers and other legacy media as well.

Some top metro papers like those in Seattle, Milwaukee and Tampa Bay determined several years ago that even as newsroom cuts become a necessity, investigative capacity should be maintained or even expanded. When Gannett was doing research as it introduced paywalls at its 80 community papers last year, it found that investigations were the top of list of “passion topics” readers were willing to pay for.

Impact gets identified and celebrated in the Pulitzers and other contests, but I am not so sure there is a reliable way to assess the volume and quality of the investigative work, if any, that your hometown paper still provides.

Not every published investigation is a gem. We have all been asked to read lengthy pieces that reflect efforts but did not come up with all that much. Also, there is a genre that rides along with law enforcement or government auditing work, creating an impression of impact but a bogus one. And my antennae are up for financial exposes that sometimes ignore basics of risk and fiduciary obligation.

Having done some, directed some and read several Russian novels worth of investigative projects, I think there is a common, sweet-spot design to the best of them. You need to identify and document a problem of some consequence (bigger than the typical TV I-team effort to help a guy get some shoddy construction fixed).

The problem should do more than leave the reader shaking his head and saying that’s unfortunate; it should be actionable. Good investigations mobilize a level of citizen awareness and often indignation. Then something happens as a result. That something can be governmental or non-governmental or both.

For instance, the recent Tampa Bay Times/Center for Investigative Reporting project identifying America’s Worst Charities is likely to result in tougher regulations in a number of states. But it should also motivate prospective donors to do a little due diligence on where the money goes before responding to a heart-tugging appeal.

Lewis said in the e-mail interview that newspapers are 50 years into evaluating the impact of investigative reporting, particularly when it changes “public laws or policies.” His framework might be less interesting to them except for “the intriguing issue” of collaborations between profits and non-profits like his joint creation of an investigative unit for the Washington Post with a Ford Foundation grant.  If those collaborative efforts grow, the for-profits will be pulled into the assessment discussion, he said.

The big impact of spectacular projects like the Boston Globe’s investigation into pedophile priests or The Washington Post’s investigation into shameful conditions at Walter Reed Hospital are self-evident. But a more modest everyday level of holding governments and other powerful institutions accountable is equally essential.

Lewis is a serial entrepreneur in launching investigative units and support organizations and some years ago won a MacArthur Foundation “genius” grant for his work. I think he is onto something again. Fleshing out the embryonic conversation about what works and why it matters is going to be essential to keep the resources to support investigative reporting flowing.

We have made it easy to comment on posts, however we require civility and encourage full names to that end (first initial, last name is OK). Please read our guidelines here before commenting.

  • http://www.robbmontgomery.com/ Robb Montgomery

    Sounds like a better plan than using Pulitzers and Peabody’s to measure the impact. Especially if investigative reporting will be increasingly funded by donors who need data to measure results.

  • Edward Ericson Jr

    Ralph Nader has a rap on this a couple decades back. He noticed that, after the glory years of the ’60s and ’70s, even really good investigative journalism stopped producing the kinds of results (i.e. indictments, seat belt laws, etc.) that had earlier been routine.

    The problem gets worse as the media landscape is atomized.

    So to measure impact you first must do survey research. Do the people actually know the story? (And realize, by “the people” I mean a fairly narrow slice of the public: people who actually read).

    There is a threshold–it might be 10, 15 percent of professional people; maybe it’s as small as five assistant prosecutors–after which a story might get some traction.

    It is then and always the media outlet’s job to pound that sucker down. I think it was Jimmy Breslin who said something like, (paraphrasing), the job is not to talk to the powerful bully, and let him have his say, and then talk to the victim and let him have his say, and then write it up nice and even. The job is writing the story over and over again, louder, until somebody does something about it.

    That sort of thing takes time and money–and patience. Circa 1992 I wrote a story about the High Sheriff who shook down his deputies–who he had sole power to hire and fire–for cash for his Vegas trips and whatnot. The sheriff had the last patronage system in the state, so the State’s Attorney (a major Democratic Party legacy), was not excited about investigating.

    I mentioned this every couple weeks. Drove my editor to distraction. The SA called me up all pissed off at home one Sunday morning. It was absolute fun.

    It took about three or four years to get the indictment (thanks to the Postal Inspectors), two more after that to get the Sheriff system abolished.

    That’s how it goes.

    Non-profits are always inventing metrics and shuffling paper around. News outlets will probably do it too. But, as with drug treatment, homeless “wrap around” services and other do-gooder-type, non-profit endeavors, inventing new annual metrics to rut more grants will not improve the provision of services.

  • King-Stanley-Krauter

    There has been many news articles written about our tax laws since the reforms of 1986 and NOTHING was accomplished by those reports. All of the pre-crisis journalism on the housing bubble and subprime mortgage fraud accomplished NOTHING because it was ignored by politicians and regulators because it was ignored by voters. And the reforms about our too big to prosecute banks has and will accomplish NOTHING until reporters start communicating like teachers instead of entertainers.
    But no one in the journalism profession is interested in their failures to communicate. I say that because it would be so easy to solve. Consider the tax laws. If every daily newspaper had just done a one page report on our tax laws every year on April 15, most voters would have become as mad as most voters do when their child brings home too many report cards in a row. And their anger would grow stronger and stronger every year that nothing was done. Then our elected officials wouldn’t have been able to avoid doing something because their failure was impossible to not see.
    With the news media’s obsession for writing the first draft of history, however, the do nothing deeds of our politicians are forgotten in the white noise of daily information. But reporters dont want to look at why they are failing to accomplish anything with so much of their hard work. I say that because they don’t care enough to overcome this problem by writing an annual one week second draft of history that could also work like the report cards that teachers use for rewarding and punishing their students. Repoters are only interested in entertaining themselves by writing the first draft of history..

  • Nader

    invstigative? I guess that doesn’t have much impact. Copy editors might, though.