What journalists can learn from scientists and the scientific method

I don’t need to tell you that the Scientific Revolution kickstarted the modern age. Kevin Kelly lists the discovery of the scientific method alongside the invention of libraries and the printing press as “meta-changes” — the evolution of evolution, changes in the ways things change.

The scientific method — the foundation of modern science — was a way to systematize knowledge production. Once discovered, it became a discovery-producing machine. Humankind had developed a way to both ask better questions about the nature of the world and to continually find better answers to those questions. Soon, the tools of this discovery-producing machine began to improve; we devised techniques such as double-blind studies and placebos. And over time, we arrived at where we are today, having landed on the moon, split the atom and cured smallpox.

But the scientific method isn’t a single thing. It’s a collection of conventions and best practices, rigorously applied. While the need for journalism and the demands on it differ in many ways from those of science, it is a discipline that — like science — seeks truth. There is no “journalistic method” to rival the scientific one. But in that body of conventions and practices, there’s much that journalists should emulate.

This is an ancient argument. Just last week, Robert Niles made a very similar case in the Online Journalism Review. And my understanding of these thoughts is built on a century of thinking from folks like Walter Lippman and the University of North Carolina’s Philip Meyer, who’ve been banging this drum for many, many years. But with the rise of the Web, there have never been more opportunities to put this thinking into practice, to create models of journalism truly informed by the science. Here are a few places we could start.

Making more accurate predictions

Part of the genius of the scientific method is that it tests how true a hypothesis is by tracking how well it predicts future outcomes. Your theory is only as good as its ability to predict the results of a trial.

We journalists like to think of ourselves as describing the present as faithfully as we can. So journalism contains no such predictive test. Yet so much of journalism involves collecting and assessing claims about the future. Take a typical story, plucked at random from a July 2009 front page of politics news aggregator Memeorandum, conveying the fears of a group of state governors that the plans to overhaul the nation’s health care systems would foist unpaid obligations on the states.

These claims are eminently trackable. In fact, PolitiFact is already doing some of this. The site’s ingenious Obameter and GOP Pledge-O-Meter systematically follow up on one common type of future-claim: a pledge that a politician will do something once elected. As the 2012 election approaches, these tools are becoming fantastic barometers for voters to assess how well incumbent candidates delivered on their promises.

Imagine if other types of future-claims — e.g. “this bill will spur hiring among small-business owners,” “if we don’t strengthen the levees, a hurricane could be catastrophic for our city” — were tracked and assessed in as vigorous and organized a fashion. Journalists would have more of an incentive to prompt their sources for testable claims. The public would have a framework for assessing sources’ credibility.

Aside from the claim-tracker approach to predictions, though, there’s a lot to be said for journalism that accurately anticipates news events. If we’ve done a thorough and accurate job of describing the present, our work should leave our users better prepared for the future.

One of my favorite works of journalism is James Fallows’ November 2002 cover story in The Atlantic Monthly, “The Fifty-First State?” In it, Fallows reports with amazing foresight the chain of events likely to occur if the U.S. were to invade Iraq. Almost nine years later, the story reads like a now-familiar catalogue of major issues the U.S. encountered after invading Iraq in March 2003, five months after the article was published. “The Fifty-First State?” won a National Magazine Award in the Public Interest category in 2003. It makes me wish there were a Pulitzer Prize for journalism from five or more years ago that presaged something important about the world we live in today.

Borrowing the concept of “reproducibility”

We use the term “transparency” a lot in conversations about journalistic ethics. It’s one of the key values of our profession: journalists should give the public the best tools they can to evaluate their work, including as much information as possible about how the work was conducted, or any relevant disclosures that might affect its credibility.

Science has a neighboring concept worth borrowing: “reproducibility.” A scientist’s findings should be both consistent enough and described well enough that another independent scientist can repeat the research and achieve a very similar outcome.

For many years, UNC’s Meyer has encouraged us to do away with the common, oft-derided understanding of journalistic objectivity that prompts us to ask questions like, “Is the story objective?” “Traditionally,” Meyer once said, “[objectivity] has sort of meant you sprinkle your ink around to lots of different groups and hope that it comes out even or fair. I think we ought to emphasize objectivity of method. That’s really what scientific method is.”

I think “reproducibility” is another way to get at what Meyer’s describing. Are the conclusions embedded in a work of journalism reproducible? Has the reporting been thorough, fair and accurate enough that another reporter digging into the same issue would reach a similar outcome? If someone else had tried to replicate Fallows’ inquiry into the aftermath of an invasion of Iraq, would they describe a similar future?

I especially like the second element of the test of reproducibility. Does the story describe its methods and inputs well enough that someone else could repeat them? Since the rise of the Web, this has been a growing part of the ethic of investigative journalism. I now expect whenever I see a big data project from a news organization that I can inspect the data myself and test what the reporters have told me. ProPublica has probably taken this ethic the furthest that I’ve seen; the site has published “reporting recipes” that lay out the steps to reproduce its work at a local level.

I would love for this to become a part of all journalism, not just investigative and data journalism. If all news stories were reported with an eye toward reproducibility, we’d be even warier about using anonymous sources. We’d be better about describing not just what our sources said, but where and how they said it. This is a pet peeve of mine. There’s a material journalistic difference between “so-and-so said” and “so-and-so said to me over drinks yesterday evening” and “so-and-so said to a group of reporters in a conference call for the press.” Only the latter versions give some indication of how one might go about recreating the work.

The beauty of process

In my work on NPR’s Project Argo, I have the pleasure of working with several terrific science reporters, at least one of whom is a bona fide scientist herself. I feel as though I now have a better sense of some of the tensions between scientists and journalists.

Science places much more of an emphasis on processes. Journalism places much more of an emphasis on outcomes. To journalists, the process of scientific discovery is tortuously incremental. We want to push all of those double-blind, randomized, peer-reviewed, clinical bits out of the way and get straight to the delicious creme filling — “NEW RESEARCH COULD LEAD TO INVISIBILITY CLOAK!!” (I particularly love the phrase “could lead to.” You can tuck so much under that rug. Obligatory link to “the science news cycle.” Obligatory link to satirical news article about scientific papers.)

Journalists think in discrete stories. As in, “I’m finished with this story. Onto the next.” We often aim to produce these polished gems of Aristotelian narrative, bearing arresting ledes, explosive kickers and genuine catharsis somewhere in the midst. In science, the continued journey toward greater knowledge is an unending quest. Scientists spend entire careers advancing the state of knowledge in their field, not whizzing from discovery to discovery, but gradually pursuing an ever-greater understanding.

The ability to turn the process of reporting into a compelling, unending story of its own is becoming an increasingly vital journalistic skill. I encourage the journalists I work with to think in streams, not just discrete stories. Your feed is more important than any single post, and a subscriber is more valuable than a visitor. If you can hook people into your ongoing quest for greater understanding of a beat or topic, you wield a much more powerful asset than the crowd of random people who happen to find one of your stories.

Paul Ford recently wrote a beautiful, much-forwarded essay for New York Magazine about the rising importance of the stream. Here’s a taste:

Social media has no understanding of anything aside from the connections between individuals and the ceaseless flow of time: No beginnings, and no endings. These disparate threads of human existence alternately fascinate and horrify that part of the media world that grew up on topic sentences and strong conclusions. This world of old media is like a giant steampunk machine that organizes time into stories. I call it the Epiphanator, and it has always known the value of a meaningful conclusion. The Epiphanator sits in midtown Manhattan and clunks along, at Condé Nast and at the Times and in Rockefeller Center. Once a day it makes a terrible grinding noise and spits out newspapers and TV shows. Once a week it spits out weeklies and more TV shows. Once a month it produces glossy magazines. All too often it makes movies, and novels.

At the end of every magazine article, before the “■,” is the quote from the general in Afghanistan that ties everything together. The evening news segment concludes by showing the Secretary of State getting back onto her helicopter. There’s the kiss, the kicker, the snappy comeback, the defused bomb. The Epiphanator transmits them all. It promises that things are orderly. It insists that life makes sense, that there is an underlying logic.

These two concepts — the Epiphanator and the stream — can and should live alongside each other. My buddy, co-blogger and fellow Poynter alum Robin Sloan, calls the interplay between the two “stock and flow.”

What we call “media literacy” depends on the public taking an interest in the particulars of the journalistic process, not just the stories it produces. A fact that a reporter has taken great pains to prosecute and verify should hold more weight than a soundbite delivered with plenty of pizazz and zero evidence to support it. But to make that distinction, we need to present the process. And we need to make it as arresting as the outcome.

I’ll appeal to that immortal James Fallows article one last time. To quote myself:

Among the article’s most notable characteristics is Fallows’ willingness to show his work — the story begins with a remarkable catalogue of Fallows’ process and assumptions. Almost everybody quoted is on the record (side note: just think of the thousands of inches of anonymously sourced stories that totally got it wrong right around this time), and we see Fallows’ perspective shift as the piece progresses. By putting all that in there, Fallows makes the story accessible, engaging, and deeply informative, not overly reflective or self-indulgent.

Journalism isn’t science, and science isn’t perfect. In fact, there are many ways that the field of science is falling behind journalism in adapting to changes in our society. But I’ve only started to scratch the surface of how journalism can build on the practices science has evolved. I’m really interested to hear how journalism might benefit from concepts analogous to peer review and theory construction.

So, keeping with Lippman’s exhortation that we approach journalism in “the scientific spirit,” let’s make this an ongoing conversation, not an end point. I hope to continue this discussion in a SXSW session I’ve pitched with my friend Gideon Lichfield, a journalist with the Economist who has two degrees in the philosophy of science. And I invite you to share your thoughts with me in the comments section of this story.

We have made it easy to comment on posts, however we require civility and encourage full names to that end (first initial, last name is OK). Please read our guidelines here before commenting.

  • http://www.facebook.com/lorenzo.mannella Lorenzo Mannella

    Thank you for this illuminating article. I am writing a thesis in science journalism, and this piece has given me some unexpected inspiration. I hope your conversation will go on further.
    Kind regards,

  • Carrie Brown

    Great piece. This is something the book Elements of Journalism addresses
    as well and that we talked to many newsrooms about when I worked for
    Committee of Concerned Journalists. I hope to see the conversation

  • http://pulse.yahoo.com/_PGQOHY2VQGD5Q4HRY4K5QYQTWA Katharine Whitley

    I just got a $829.99 iPad2 for only $103.37 and my mom got a $1499.99 HDTV for only $251.92, they are both coming with USPS tomorrow. I would be an idiot to ever pay full retail prices at places like Walmart or Bestbuy. I sold a 37″ HDTV to my boss for $600 that I only paid $78.24 for. I use GetCent.com

  • Anonymous

    The best thing I’ve read on “Journalism” in a long while. Worth the click and a SLOW read. http://gaihapdan.com

  • 1 1

    News world: Many others could also see that ghastly folly in the making, but the
    “establishment” (“insider”-accessing) press held onto its privileged
    position in the echo chamber of MSM.

  • 1 1

    News world:  Many others could also see that ghastly folly in the making, but the
    “establishment” (“insider”-accessing) press held onto its privileged
    position in the echo chamber of MSM.

  • Anonymous

    welcome to: http://www.fullstores.com

    The website wholesale for many kinds of fashion shoes, like the nike,jordan,prada,****, also including the jeans,shirts,bags,hat and the decorations. All the products are free shipping, and the the price is competitive, and also can accept the paypal payment.,after the payment, can ship within short time.

    free shipping

    competitive price

    any size available

    accept the paypal


  • Anonymous

    Novelty means news for journalists, but novelty for scientists is
    inherently suspect. The result is a completely different conceptual
    scheme for understanding truth. Peer-reviewed publication is usually
    sufficient for a claim to make it into the news, but peer-review is
    merely a threshold for having one’s findings discussed in science. The
    fact that so much peer-reviewed material turns out to be flawed should
    be a wake up call for journalists. Something that’s new in science is
    never as justifiable as the weight of evidence. 

  • http://www.facebook.com/people/Shop-HerveLeger/100002628362011 Shop HerveLeger


    If you are not satisfied within 30 days can be returned to us.
    Shopping at wowdress.co.uk is easy,convenient and safer.

  • Anonymous

     Sounds like you are advocating  a synthesis of precision journalism and narrative journalism. Good idea! I used to agree with Ev Dennis and Bill Rivers (“Other Voices: the New Journalism in America,” 1974) that precision journalists and narrative journalists were natural enemies. But they don’t have to be. Both forms give framing and structure to the endless flow of facts. As it happens, I am preparing a lecture on the value of a journalism that would combine these two forms that were “new” in the 1970s. The Information Age creates a need for such intense management of data.

  • Anonymous

    Great piece, with useful links to same. Some basic journalism practice elements I have taught include checking sources’ comments with (and not just against) other sources (rather than just listing them alongside each other for contrast or support). Encouraging a debate / discussion at pre-publication stages allows reporters to get closer to the “truth” (as far as it can be determined given time and other constraints), and it allows aspects of the reportorial process to be revealed to the reader, to build credibility and expose possible limits or information advantages sources (and the reporter) display. +1 for promoting an award (Puliitzer or other) for journalism that, in retrospect, provided a fruitful pathway for greater understanding, a la the estimable James Fallows’ prescient 2002 look at the looming Iraq disaster. Many others could also see that ghastly folly in the making, but the “establishment” (“insider”-accessing) press held onto its privileged position in the echo chamber of MSM. I would argue that the badly flawed work of the “leading” voices in MSM prepared the way for the growing skepticism about MSM and, in tandem with technological change, propelled us into the current world of myriad claims to truth, with its lagging but hopefully eventually effective corrective of fact-checking sites (btw, in this vein, don’t ignore Newstrust, which just went open-source to bring still more voices into the equation). Anyway, thanks for an excellent, provocative piece. I will forward it copiously. 

  • Anonymous

    I came to journalism from anthropology and have been teaching for four years that one of our emerging paths forward is recognizing journalism as a social science of current events. Understanding the theory and methods of journalism as practiced now and how they could be applied through alternative presentation of information and analysis naturally leads reflective thinkers to a processualist approach, which is what you are identifying as scientific method here. The application of a research question or hypothesis as a framework instead of a lede and nutgraf and story shape will change our views of storytelling and narrative as the center of conveying information in the journalistic setting. Understanding and approaching current events from a research perspective over a storyteller’s perspective fundamentally alters our ability to relate and interpret these events. Thanks for your post, Mat, and I hope your panel succeeds. I’d welcome an opportunity to be a part of that extended conversation. 

  • http://twitter.com/datacanlas datacanlas

    I suppose the practice of creating streams and deeply-researched journalistic stories can happen when the business of producing mass media is less suffocating. Because news is also treated as a product to be sold, journalists need those visceral hooks and strong endings and “exposes” so that their newspapers/radio programs/news shows will sell. The concept of scoops, though favored by the business, isn’t at all favorable to the practice of journalism. I’ve met some journalists who make it their goal to outscoop the competition. That it’s come to that is a sad thing — if they can’t see there’s a bigger responsibility to the audience, the citizens, the country, how can they see the necessity in building up knowledge and creating their streams? Paradigm shifts are in order, I guess, and I hope journalists see the beauty and elegance in this philosophy.

  • http://www.facebook.com/profile.php?id=628233825 Trevor Butterworth

    Novelty means news for journalists, but novelty for scientists is inherently suspect. The result is a completely different conceptual scheme for understanding truth. Peer-reviewed publication is usually sufficient for a claim to make it into the news, but peer-review is merely a threshold for having one’s findings discussed in science. The fact that so much peer-reviewed material turns out to be flawed should be a wake up call for journalists. Something that’s new in science is never as justifiable as the weight of evidence.