Claim: Krugman is top prognosticator; Cal Thomas is the worst

Romenesko Misc.
A Hamilton College class and their public policy professor analyzed the predicts of 26 pundits — including Sunday morning TV talkers — and used a scale of 1 to 5 to rate their accuracy. After Paul Krugman, the most accurate pundits were Maureen Dowd, former Pennsylvania Governor Ed Rendell, U.S. Senator Chuck Schumer (D-NY), and former House Speaker Nancy Pelosi. “The Bad” list includes Thomas Friedman, Clarence Page, and Bob Herbert.

Press release

Pundits predict no more accurately than a coin toss
Krugman tops, Cal Thomas bottom of accurate predictors, according to study at Hamilton College

CLINTON, N.Y. – Op-ed columnists and TV’s talking heads build followings by making bold, confident predictions about politics and the economy. But rarely are their predictions analyzed for accuracy.

Now, a class at Hamilton College led by public policy professor P. Gary Wyckoff has analyzed the predictions of 26 prognosticators between September 2007 and December 2008. Their findings? Anyone can make as accurate a prediction as most of them if just by flipping a coin.

Their research paper, “Are Talking Heads Blowing Hot Air? An Analysis of the Accuracy of Forecasts in the Political Media” will be presented via webcast on Monday, May 2, at 4:15 p.m., at www.hamilton.edu/pundit. The paper will also be available at that address at that time. Questions during the presentation can be posed via Twitter using #hcpundit.

The Hamilton students sampled the predictions of 26 individuals who wrote columns in major print media and who appeared on the three major Sunday news shows – Face the Nation, Meet the Press, and This Week – and evaluated the accuracy of 472 predictions made during the 16-month period. They used a scale of 1 to 5 (1 being “will not happen, 5 being “will absolutely happen”) to rate the accuracy of each, and then divided them into three categories: The Good, The Bad, and The Ugly.

The students found that only nine of the prognosticators they studied could predict more accurately than a coin flip. Two were significantly less accurate, and the remaining 14 were not statistically any better or worse than a coin flip.

The top prognosticators – led by New York Times columnist Paul Krugman – scored above five points and were labeled “Good,” while those scoring between zero and five were “Bad.” Anyone scoring less than zero (which was possible because prognosticators lost points for inaccurate predictions) were put into “The Ugly” category. Syndicated columnist Cal Thomas came up short and scored the lowest of the 26.

Even when the students eliminated political predictions and looked only at predictions for the economy and social issues, they found that liberals still do better than conservatives at prediction. After Krugman, the most accurate pundits were Maureen Dowd of The New York Times, former Pennsylvania Governor Ed Rendell, U.S. Senator Chuck Schumer (D-NY), and former House Speaker Nancy Pelosi – all Democrats and/or liberals. Also landing in the “Good” category, however, were conservative columnists Kathleen Parker and David Brooks, along with Bush Administration Treasury Secretary Hank Paulson. Left-leaning columnist Eugene Robinson of The Washington Post rounded out the “good” list.

Those scoring lowest – “The Ugly” – with negative tallies were conservative columnist Cal Thomas; U.S. Senator Lindsey Graham (R-SC); U.S. Senator Carl Levin (D-MI); U.S. Senator Joe Lieberman, a McCain supporter and Democrat-turned-Independent from Connecticut; Sam Donaldson of ABC; and conservative columnist George Will.

Landing between the two extremes – “The Bad” – were Howard Wolfson, communications director for Hillary Clinton’s 2008 campaign; former Arkansas Governor Mike Huckabee, a hopeful in the 2008 Republican primary; former House Speaker Newt Gingrich, a Republican; Sen. John Kerry of Massachusetts, the Democratic nominee for president in 2004; liberal columnist Bob Herbert of The New York Times; Andrea Mitchell of NBC; New York Times columnist Tom Friedman; the late David Broder, former columnist for The Washington Post; Chicago Tribune columnist Clarence Page; New York Times columnist Nicholas Kristof; and Hillary Clinton.

The group also found a link between conditional predictions and accuracy, that is, a prediction that was conditional (“If A, then B”) was less likely to be accurate. Finally, those prognosticators with a law degree were more likely to be wrong.

We have made it easy to comment on posts, however we require civility and encourage full names to that end (first initial, last name is OK). Please read our guidelines here before commenting.

  • Anonymous

    You make, I must admit, an excellent point.  Perhaps I should say it is the product of some ignorant professionals, lol.  

    On reflection, my understanding of the Austrian school is lacking.  For example, I know they hate central banks because they consider them an irresponsible way to add to the monetary supply.  But…what is their alternative?  Is it really going to the gold (or some other commodity) standard, which seems rather like trusting chance instead of taking a chance with trust? Or is it something else, and I’m just not well versed enough in their theories? 

    Thanks for the food for thought. My next rainy day in a good library or downloading some books will mean some attention paid to my scholarly deficit in this area.  

  • http://www.facebook.com/profile.php?id=709397300 Anonymous

    The truth is that big government leads to national bankruptcy.

    Queue the references to Norway and Sweden..

  • http://www.facebook.com/profile.php?id=709397300 Anonymous

    Be careful about the illusion of economic expertise. Economics is not a hard science, and therefore it is only accountable to peer opinion, which can be affected by prevailing dogma and conventional wisdom.

  • http://www.facebook.com/profile.php?id=12910958 Tyler Smith

    Actually, if you look at page 55 and 56 of the report itself, they go over this specifically. They ran another test taking out election predictions entirely, and still got the same result.

    “However, this large statistically significant coefficient signaled a red flag for us – it seems unlikely that Democrats would beat out Republicans by such a large margin. Our first instinct was that the election outcome (with Democrats and Barack Obama winning by large margins) could have had a confounding effect on the partisanship results. Democrats predicted optimistically that their candidates would win election, and the election swung in their favor, thus making their rhetorically-driven predictions correct. In order to decipher whether or not the election had skewed our results, we decided to run the regression again, this time excluding all the predictions having to do with the November election. Partisanship was once again statistically significant, showing that the liberal prognosticators had actually predicted more accurately than their Republican counterparts during our time period.”

  • http://twitter.com/Mario_Serg Mário Sergio
  • http://profiles.google.com/quantumtuba Nathan Goodman

    To be clear, there are PhD economists who espouse the Austrian economics you deride, and the Austrian school has produced at least one Nobel Prize Winner (Hayek). Now, there are good arguments against their conclusions and methodology, and Krugman has some, but to simply dismiss it as the product of ignorant amateurs is an invalid argument.

  • http://pulse.yahoo.com/_QFCBHMT5P2GROKEKWFSH2ZMFQE Final Point

    Conservatism is based off lies. Conservatism is the domination of society by it’s top 1%, ie. aristocrats. It is the maximum amount of domination a society will allow at any one point.

  • Anonymous

    You haven’t even read the report’s methodology. While what you’re saying is certainly worth considering, your claim that it would be the total opposite before the 2000 election is nothing but blind speculation. It’s not like you can count the same prediction multiple times. We don’t even know what portion of predictions were “who will win”; questions about things like the economy or foreign affairs would not be significantly influenced by the election.

    Real economists, and people who listen to real economists (Krugman, some other liberal commentators) are going to do a better job of evaluating the economy than people who use “common sense” or listen to economists who virtually always work for conservative/libertarian think-tanks, like the Heritage Foundation or the Cato Institute.

    The idea that the market is always better/more efficient than the government is completely false. Have you ever heard any evidence aside from “common sense” or someone talking about the DMV/USPS? Most economists know this, but conservative pundits/politicians still keep repeating it. Industries/societal needs with certain attributes generally have their needs met better by government, and there is plenty of data to support this. A few of these attributes are inelastic demand, involving a public good, and naturally poor competition. Health care is a perfect example of an industry that serves consumers better when government-run. Low costs and healthier citizens in a multitude of other countries show this to be the case.

  • Anonymous

    You haven’t even read the report’s methodology. While what you’re saying is certainly worth considering, your claim that it would be the total opposite before the 2000 election is nothing but blind speculation. It’s not like you can count the same prediction multiple times. We don’t even know what portion of predictions were “who will win”; questions about things like the economy or foreign affairs would not be significantly influenced by the election.

    Real economists, and people who listen to real economists (Krugman, some other liberal commentators) are going to do a better job of evaluating the economy than people who use “common sense” or listen to economists who virtually always work for conservative/libertarian think-tanks, like the Heritage Foundation or the Cato Institute.

    The idea that the market is always better/more efficient than the government is completely false. Have you ever heard any evidence aside from “common sense” or someone talking about the DMV/USPS? Most economists know this, but conservative pundits/politicians still keep repeating it. Industries/societal needs with certain attributes generally have their needs met better by government, and there is plenty of data to support this. A few of these attributes are inelastic demand, involving a public good, and naturally poor competition. Health care is a perfect example of an industry that serves consumers better when government-run. Low costs and healthier citizens in a multitude of other countries show this to be the case.

  • Anonymous

    You haven’t even read the report’s methodology. While what you’re saying is certainly worth considering, your claim that it would be the total opposite before the 2000 election is nothing but blind speculation. It’s not like you can count the same prediction multiple times. We don’t even know what portion of predictions were “who will win”; questions about things like the economy or foreign affairs would not be significantly influenced by the election.

    Real economists, and people who listen to real economists (Krugman, some other liberal commentators) are going to do a better job of evaluating the economy than people who use “common sense” or listen to economists who virtually always work for conservative/libertarian think-tanks, like the Heritage Foundation or the Cato Institute.

    The idea that the market is always better/more efficient than the government is completely false. Have you ever heard any evidence aside from “common sense” or someone talking about the DMV/USPS? Most economists know this, but conservative pundits/politicians still keep repeating it. Industries/societal needs with certain attributes generally have their needs met better by government, and there is plenty of data to support this. A few of these attributes are inelastic demand, involving a public good, and naturally poor competition. Health care is a perfect example of an industry that serves consumers better when government-run. Low costs and healthier citizens in a multitude of other countries show this to be the case.

  • http://profiles.yahoo.com/u/GQQIJ5U6H3BDGSXIMPCJSHHUPY Pee Pee

    “Thomas Friedman” is an anagram for “Wrong About Everything”

  • Anonymous

    Maureen Dowd is liberal? Tell it to Al Gore.

  • Anonymous

    Interesting but Sunday morning talking heads are far behind events. Far better to rely on the international media. Do a survey of Gwynne Dyer and get back to me.

  • http://www.facebook.com/profile.php?id=1428470021 Jonathan Hendry

    Their George Will data is messed up. They have him with his total # of predictions as 17, but the Right/Wrong/Hedged counts don’t add up. The actual # of predictions is 61, which is mentioned elsewhere in the text of the report.

    Makes me wonder if his score/ranking was calculated with the correct # of predictions, or the 17.

  • Anonymous

    The guy with a PhD in economics, and a Nobel prize in economics, who writes mostly about economics, is the most accurate? There’s a shocker.

    Nobody would presume to tell a surgeon how he should do his or her job. Or a physicist, or an airline pilot. But in matters of economics, people who have never cracked open a freshman level textbook on the subject think they can figure it all out by using “common sense”, and likening federal fiscal policy to their household budget.

    Which leads to wacky ideas like abolishing the Fed and returning to the gold standard – it is astounding that anyone espousing such ignorance can be taken seriously, but that’s politics for you.

  • Anonymous

    The guy with a PhD in economics, and a Nobel prize in economics, who writes mostly about economics, is the most accurate? There’s a shocker.

    Nobody would presume to tell a surgeon how he should do his or her job. Or a physicist, or an airline pilot. But in matters of economics, people who have never cracked open a freshman level textbook on the subject think they can figure it all out by using “common sense”, and likening federal fiscal policy to their household budget.

    Which leads to wacky ideas like abolishing the Fed and returning to the gold standard – it is astounding that anyone espousing such ignorance can be taken seriously, but that’s politics for you.

  • http://www.facebook.com/larry.nocella Larry Nocella

    Gotta say this doesn’t surprise me. Generally speaking, conservatives value faith over facts, and therefore they are more likely to be wrong.

    and p.s. I totally agree William Kristol should have been included to show just how wrong he regularly is.

  • http://twitter.com/jase23 Jason Miller

    No, nobody said it would be easy.

  • Anonymous

    . . . .and liberals were wrong between 2008-2011?

  • http://twitter.com/axomamma karen marie

    Yeah, because if the period studied was between 2000 and 2007, liberals would have fared even better, because conservatives were wrong about everything during that time period.

  • http://twitter.com/LIBrods Lisa Brodsky

    I think you hit this one right on the head. Liberal prognosticators were more accurate because they predicted what they and their followers wanted, and more liberal wishes than conservative wishes came true during that time period. A much more useful study would see how often these pundits were right when they predicted something against their own party’s interests… but that probably never happens.

  • Anonymous

    In unrelated news, a study of predictions in January-February of 2011 shows that Green Bay has the most accurate prognosticators of any town in the United States. President Obama has appointed Aaron Rodgers as Chief Economic Officer. http://en.wikipedia.org/wiki/Super_Bowl_XLV

  • Anonymous

    I’m a little skeptical of the results only because of the short period of time studied. However, this is something I’ve long wished for because these pundits have so much influence on our government and media. George Will often writes about how bad teachers should be fired and I’ve always silently replied “So shouldn’t we also fire bad pundits?” George, it appears you’re on shaky ground here.

  • Anonymous

    I think the obvious flaw in this is that this was done during 2007 – 2008, so most predictions would be about the presidential election. In most presidential elections, liberals tell their readers that the Democratic party will win and vice versa to bolster support. The fact that a Democrat won would tend to make liberals more correct. If we reran this study in 1999-2000 or even 2009-2010, the results would have been the opposite. Was this really not obvious to the journalist here?

  • http://www.facebook.com/people/Christopher-Crittenden/753735580 Christopher Crittenden

    I wonder if their results would have been the reverse if they would have looked at different years. Are liberals more likely to be correct when the economy is going one way and conservatives more likely to be correct when it’s going the other way?

  • Anonymous

    How can you “sample” (their language) predictions without introducing bias? Maybe the original paper addresses this issue, but there’s not a word in this report. “Predictions” can be broad and rhetorical opinion (“‘the US is turning into a fascist state”) , or obvious (“government spending will increase next year”). The temptation to pick easy ones and go easy on the evaluation for your favorite columnists makes this into another silly exercise in self-aggrandizement by the academic left.

  • Patrick DeBurgh

    A Liberal has the truth bias.

  • http://pulse.yahoo.com/_7KUWW2NDDTAFQS5NLCA2OKNTEQ Kyle

    The truth has a liberal bias.

  • pragmatism

    good thing megan mcarglebargle isn’t classified as a “pundit” or she would have been dead last. or maybe her defenses of “technically true but collectively nonsense” or “that’s not what i meant when i wrote that, i actually meant the opposite” would have saved her.

  • Anonymous

    The fact that anyone can match a pundit merely by flipping a coin brings to mind Calvin Trillon’s great appellation regarding Sunday talking heads. “Sabbath Gasbags” he called them.

  • http://profiles.yahoo.com/u/QXHUG6B22T6CKA3CRQ4EG3UOCM Laurence

    This list is totally bogus because it leaves out William Kristol of the mostly unwatched “Fox News Sunday”. WRONG-WRONG-WRONG over and over again. (Many people consider him responsible for getting John McCain to believe that he and Sarah Palin would be a winning combination).

  • http://profiles.yahoo.com/u/QXHUG6B22T6CKA3CRQ4EG3UOCM Laurence

    This list is totally bogus because it leaves out William Kristol of the mostly unwatched “Fox News Sunday”. WRONG-WRONG-WRONG over and over again. (Many people consider him responsible for getting John McCain to believe that he and Sarah Palin would be a winning combination).