Maybe you guys are misssing some point...
[EDITORIAL NOTE: THIS COMMENT WAS ORIGINALLY POSTED ANONYMOUSLY IN PORTUGUESE, AND REPLICATED IN ENGLISH HERE TO EASE DICUSSION.]
Teach the World Bank statistics. Cult of statistical significance or IV. (Teach the World Bank statistics. Cult of statistical significance or IV.)
by Carlos Cinelli
Consider two random samples with 10 observations, drawn from a normal distribution with different means and the same variance unknown. To use a concrete example, R simulates the two samples, one from a normal with mean 5 and standard deviation 1 and the other a normal with mean 2 and standard deviation 1.
The samples resulted in the following statistics:
Sample mean: 5
Sample standard deviation: 0.8
The 95% confidence interval: 3.4 to 6.6
Sample mean: 2.6
Sample standard deviation: 0.7
The 95% confidence interval: 2.6 to 4.0
Note that the confidence intervals intersect. The lower limit of sample 1 is 3.4 and the upper limit of sample 2 is 4.0.
This means that the difference between the sample means is not statistically significant at 5%?
Not by far!
Making a t test for the difference between the two means you get a statistically significant result. Even if you did not know that the variances were equal, the Welch t test gives us a range of 95% confidence interval for the difference between the averages between 1.8 and 3.2.
Now imagine that these data were GDP growth, ie, a group has sample average growth of 5% and others 2.6%. If you compare the confidence intervals, you might tend to say that the two groups have no growth "different" ... when, in fact, a proper test classic differences between means indicates a difference between 1.8 and 3.2 points percentage, very important in terms of economic growth!
But this error happens?
Yes, the World Bank. On Econbrowser on the controversy Heinhart and Rogoff, Chinn released this graph relating the average growth and percentage of public debt to GDP ratio. The bars are the mean and the black line represents the 95% confidence level.
debtgdpgrowth.pngNote that although the average growth of countries with high debt (over 90% of GDP) to be much lower than the average of the other, the confidence intervals intersect. This led the staff blog of the World Bank said that "[...] the confidence intervals of all three bins above the 30 percent debt / GDP threshold Also overlap. On this (admittedly crude) basis, then, any claim que a 1 percent growth differential over a decade compounds is simply overstating the case made by the date. "
This is wrong, the simple fact of the ranges of 95% confidence cross does not mean anything, even if you thought that statistical significance was the relevant point here. As we saw in the previous example, with a super simple example, the confidence intervals can intersect and yet the difference is "statistically significant" and, more importantly, economically relevant! Aware of the error, the authors did a PS warning for apparel and reducing the confidence interval for standard deviation, instead of two ...
Despite the play's title, it was not a "stupidity" of the World Bank. One problem I have encountered when discussing this is that, in general, people think that only "dumb" make this kind of mistake, or just "journals" crappy publish things like this. Ledo mistake ... misunderstanding about confidence intervals, statistical significance, p-values are pervasive in the social sciences, including applied work in the best journals and with the best researchers. Be in the USA, Brazil or Germany, this happens a lot and it's something we have to change.
source: source: http://analisereal.com/2013/04/19/ensinem-estatistica-ao-banco-mundial-ou-culto-da-significancia-estatistica-iv/