Oops, They Did It Again:

Smackdown Shows Why “Small Stuff” Actually Matters. A Lot.

With a new job starting this week, I thought that I wouldn’t have time to write a blog post.  That is, until my husband noticed one of our favorite Wall Street Journal columns by Carl Bialik, the “numbers guy,” this weekend.[1] Bialik highlighted the rapid-fire debate that ensued after students in an Applied Econometrics class at the University of Massachusetts were assigned the task of replicating a published study. Twenty-eight-year-old doctoral student Thomas Herndon chose “Growth in a Time of Debt,” authored by Carmen Reinhart and Kenneth Rogoff—two Harvard economists with sterling reputations—and published in the American Economic Review in 2010. The study, described by the New York Magazine “Daily Intelligencer” as “massively influential,”[2] found a strong association between increasing government debt and declining economic growth (e.g., average annual growth of only 1.7% for countries with debt exceeding 90% of gross domestic product versus 3.7% when debt was less than 30%).[1,3] Political debate today being what it is, the study has been cited repeatedly by leading conservatives to justify cutbacks in government spending.[2]

Much to his credit, Herndon wrote to the authors and requested a copy of their dataset and, after not receiving a reply initially, he repeated his query. Much to their credit, Reinhart and Rogoff sent the data, in an Excel spreadsheet, along with a note telling him to “feel free to publish whatever results” he found, according to Herndon.[2] That’s when things started to get interesting:

“I clicked on cell L51,” Herndon said, describing the incident in a recent interview, “and saw that [the authors] had only averaged rows 30 through 44, instead of rows 30 through 49.”[2] In other words, Reinhart and Rogoff had inadvertently excluded five data points (countries) from their calculations. In a reanalysis including the omitted nations, high debt was associated with 0.2% annual growth.[1] And, again with political debate today being what it is, the reanalysis has been seized upon by progressives to argue that Reinhart and Rogoff’s central thesis of a link between debt and suppression of economic growth “has been substantially weakened”[4] and "there’s no question that the austerity movement has been dealt a major blow” by Herndon.[2]

Yet, in all the hubbub over these findings—and the justifiable pride of U Mass in the good detective work done by one of its own—several critically important points were missed. And, although the desire to capitalize politically on a mistake made by someone with a different viewpoint is perhaps predictable, none of the most important aspects of this story has anything whatsoever to do with liberal or conservative politics. They don’t even have anything to do with academic competition, although that aspect of the story is interesting. They are all, in fact, about research methods. (C’mon, you had to know I’d say that—this is a research methods blog, after all.)

First, although news coverage of this debacle described it as remarkable, situations of this type are commonplace. As a research consultant and then journal editor, I met many highly talented professionals with advanced degrees—almost all of whom were knowledgeable in their fields, and none of whom had received any training in managing a dataset and data analysis to prevent calculation errors. One might believe that this training deficit is of no importance—until looking at the history of publication missteps and discovering that what happened to Reinhart and Rogoff happens all the time.

Not convinced? Peek “inside the statistical black box,” as I reported in 2006 and in Health Care Research Done Right:

·  A highly influential study on the economic effects of divorce, published by a respected Stanford University sociologist, was in the subsequent decade cited in 175 popular press stories, 250 law review articles, 348 social science articles, numerous court cases, and President Clinton’s 1996 budget. The study’s author, Lenore Weitzman, later recalled that 14 new divorce laws were passed in the state of California (the study’s setting) as a result of her finding that no-fault divorce had “disastrous” effects on women and children. But when initially reported, the study’s findings were so at variance with previously published analyses of the same research question that other social scientists began requesting access to the data. Unfortunately, Weitzman refused to provide them for many years, and no reanalysis took place until 1996—more than ten years after the original publication. Reanalysis showed that Weitzman’s widely used calculations were wrong—the result of keying errors and calculation mistakes made by one or more graduate students working under her supervision.[5,6]

·   Because of a miscommunication, a data analytics consulting firm working for Arizona’s Independent Redistricting Commission inadvertently tallied both active and inactive voters in calculating district sizes. The error was discovered three months after the commission thought its work was completed, and a local newspaper reported in April 2002 that candidates didn’t “know where to collect the signatures and donations they need to run for office.”[5]

 

·   The University of East Anglia’s Climatic Research Unit (CRU) reported in 2009 that the raw data necessary to replicate the “hockey stick” graph documenting the existence of human-caused global warming were no longer available to anyone—including the CRU. For years, the CRU had failed to respond to Freedom of Information Act requests for the data because after “cleaning” the data (using undocumented methods), CRU threw them away when it moved to a new building. Independent investigations conducted by a team of scientists and by a United Kingdom House of Commons panel reached the same conclusion: the CRU scientists were dedicated, but disorganized. The original, raw data had been irretrievably lost and could not be recovered.[6]

 If there is anything remarkable here, it is that Rogoff and Reinhart—unlike Weitzman and the CRU—provided their dataset in what was apparently (given that Herndon was able to complete his assignment) a reasonable period of time. That shouldn’t be remarkable behavior, but the history described above suggests that it is.

Thus, the sequence of events surrounding the reanalysis of Rogoff and Reinhart’s work is just one more reminder of the need to incorporate systematic and structured data quality control measures in conducting research projects. As described in Health Care Research Done Right, five relatively simple procedures will, if practiced consistently, prevent small errors from sidetracking otherwise important research.[6]

Second, as the “numbers guy” reminds us, both of these analyses of the relationship between debt and growth have limited value, because they measure association. Only. Not causation. “The findings show a link between debt and GDP,” Bialik correctly observed, “not which causes which.”[1] To assess causation, directly addressing the policy question of whether government austerity or liberal spending policies produce growth, would require an experiment in which reasonably comparable states or countries implement private-sector versus government-growth policies and measure the results. (Although outside the scope of this blog, one could argue that today’s between-state differences in economic policies are, to some extent, meeting this need, and we should measure those outcomes.)

And this brings me to my final point. For anyone to argue that any single research finding proves anything is hubris, especially when previously reported results point in a different direction. (One possible exception is when a study is truly groundbreaking, exceptionally well done, and nationally representative, such as the Rand Health Insurance Experiment.) Herndon did well, and I hope that he benefits from his willingness to peek inside the statistical black box. However, given the methodological limitations of measuring association instead of causation, neither his work nor the work that he replicated provides actionable information. There was no "major blow" here, just as there was no compelling finding in the original analysis. From a policy perspective, this was much ado about (almost) nothing.

So, for nonresearchers, the moral of this story is to pay attention to research design, remembering again the important lesson that association does not prove causation. And we researchers should hope that Herndon’s instructors encourage him to do what we all should be doing—interpreting our research findings with humility and in the context of work performed by others. Even Harvard economists.

 

[1] Bialik C. The numbers guy. Spreadsheet slips not economists’ only problem. WSJ. April 20-21, 2013.
[2] Roose K. Meet the 28-year-old grad student who just shook the global austerity movement. April 18, 2013.
[3] Reinhart CM, Rogoff KS. Growth in a time of debt. NBER working paper no. 15639. January 2010.
[4] How Thomas Herndon, a student, took on Harvard economists and won. April 18, 2013.
[5] Fairman KA. Peeking inside the statistical black box: how to analyze quantitative information and get it right the first time. J Manag Care Pharm. 2006;13(1):70-74.
[6] Fairman KA. Health Care Research Done Right: A Journal Editor Shares Practical Tips and Techniques for High Qualityand Efficiency. Denver, CO: Outskirts Press; 2012.