KS Button, JPA Ioannidis, C Mokrysz, BA Nosek, J Flint, ESJ Robinson, and; MR Muafo (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience : 10.1038/nrn3475
A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.
When you do a study to look for changes you do a “power calculation”. This is where you determine how big your groups have to be so that you can detect a specified change with the likehood of chance to see that effect.
Therefore POWER is the strength of the ability to detect change. Low POWER means it unlikely that you can find a small change. Power calculations are the corner stone of clinical trials and normally you set it at 80% power to detect change of blah, blah, blah. In this report where the information can be found to work the power of the study, it was found to be less than 40%.
However in animal studies power calculations are normally never reported (less than 2% of the time), because animals studies invariably report positive results, otherwise it would not be published. However, the results of underpowered studies are more likely to give false positive results, where the answer is wrong.
Many animal studies are not sufficiently large such that it has any meaningful “power” to find the real result.
This new study shows that studies in neuroscience (the study of nerves) are hopelessly underpowered and that means many of the claims of the next cure (of MS?), based on animals studies are likely to be a load of tosh!
We have been saying for some time that many animal studies are probably a load of old rubbish. This is one of the reasons not to report on them. However, many studies are great. Until some standards are introduced on this aspect many more of the results will be found to be written on toilet paper.