By now, the econoblogosphere has heavily skewered the Reinhart-Rogoff debt and growth study, following Herdon, Ash, and Pollin's devastating critique of their methodology. Reinhart and Rogoff have themselves responded to the firestorm of criticism, and to their credit, have openly admitted that there was indeed an error in their computations. It's also worth laying aside the fundamental causality issues, since this has been repeatedly raised by others as a major problem, and is essentially something that the authors have stressed themselves (although in their policy dialog they have been far bolder in drawing prescriptive implications based on the correlation).

In any case, one of the more robust defenses of their results does deserve some closer scrutiny. In particular, Reinhart and Rogoff have argued that the revised results, along with the UMass trio's own, continue to point to a substantially lower average growth rates among economies that exceed the 90 percent debt/GDP ratio: in their own words, "[i]t is utterly misleading to speak of a 1% growth differential that lasts 10-25 years as small."

But this again misrepresents the strength of their results. This is because we can only confidently speak of the one percent difference as being significant if, indeed, the difference is statistically significant. And, at least at the fairly standard threshold of 95 percent confidence, there is reason to believe that any such differences may be yet another chimera. While access to Reinhart and Rogoff's original data is elusive---making it impossible to definitively verify the veracity of this claim---the widely-circulated Excel snapshot provides us with some data to work with (with the added plus that standard errors of the mean calculated using this approach also use the average-of-averages weighting scheme they employ in their paper). And attempting to replicate the key figure shows that, while the means for the observations in the greater-than-90-percent bin are it is certainly lower than the other bins (see figure below), the confidence intervals of *all three bins* above the 30 percent debt/GDP threshold *also substantially overlap*. On this (admittedly crude) basis, then, any claim that a 1 percent growth differential over a decade compounds is simply overstating the case made by the data.

Source: World Bank staff calculations, from Reinhart-Rogoff data fragment.

Note: Means for each bin are simple averages of all by-country observations within each respective bin, weighted equally by country. Observations for the >90% bin include updated data for New Zealand for all years, averaging 2.58%. 95 percent confidence intervals computed from standard errors for available observations within each mean.

**Postscript: **As a point of clarification, it should be noted that confidence intervals can overlap and still show a significant difference between means; see, in particular, this paper (PDF). Confidence interval bars *can* in fact overlap even at conventional significance levels (of, say, p = 0.05), although not substantially. For the purposes of eyeballing comparisons, perhaps standard error bars are more informative, with the broad rule of thumb that standard error ranges should be separated by about half the width of the bars before differences are significant (so that overlaps would certainly indicate the absence of significance). And as can be seen in this more stringent case (see figure below), there is still overlap between the 30--60/60--90% bin, and between the 30--60/>90% bin.

Source: World Bank staff calculations, from Reinhart-Rogoff data fragment.

Note: Means for each bin are simple averages of all by-country observations within each respective bin, weighted equally by country. Observations for the >90% bin include updated data for New Zealand for all years, averaging 2.58%. Standard errors computed for all available observations within each mean.

## Comments

## Cult of statistical significance

## On the graphical representation of statistical significance

## Hi, JamusThese are not

## Are statistical comparisons of means reasonable?

## Add new comment