(The other smaller glow to the left is “associated” with pleasant thoughts of Myers Dark rum, now ridiculously expensive.)
The synaptic juices started flowing and the glow glowed after I read “Many Neuroscience Studies May Be Based on Bad Statistics” in Wired, which opens:
The fields of psychology and cognitive neuroscience have had some rough sledding in recent years. The bumps have come from high-profile fraudsters, concerns about findings that can’t be replicated, and criticism from within the scientific ranks about shoddy statistics. A new study adds to these woes, suggesting that a wide range of neuroscience studies lack the statistical power to back up their findings.
The study is “Power failure: why small sample size undermines the reliability of neuroscience” in Nature Reviews Neuroscience by Katherine Button, John Ioannidis, and several others. Best thing about that paper was a short guide to terms researchers ought to know. My favorite:
The winner’s curse refers to the phenomenon whereby the ‘lucky’ scientist who makes a discovery is cursed by finding an inflated estimate of that effect. The winner’s curse occurs when thresholds, such as statistical significance, are used to determine the presence of an effect and is most severe when thresholds are stringent and studies are too small and thus have low power.
Button and team did a meta-meta analysis of fMRI studies and the like and discovered what will be no secret to regular readers: the statistics of these works ain’t too hot. Specifically, many (most?) have very low power. “The consequences of this include overestimates of effect size and low reproducibility of results.”
They looked at 48 meta-analyses, which comprised “730 individual primary studies.” The median power was 18%. If you don’t have a feel for that, the “normal” power for medical studies is 80%+. That’s the level grant granters want, anyway. Button’s finding means half the studies are worse than anemically powered.
The Scientist quotes Hal Pashler, a psychologist at the University of California, San Diego, as saying, “This paper should help by revealing exactly how bad things have gotten.” Can’t go too much by that, because it’s standard journalistic practice to fetch a quote from somebody who didn’t write the paper (and often didn’t read it). But in this case Pashler is right.
Or maybe I’m just happy to agree with him. Here’s why I do.
Point one, I did an extensive (maybe too extensive) critique of Sam Harris’s paper “The neural correlates of religious and nonreligious belief.” One of the worst papers, in a series of bad papers, that I’ve ever read. Shoddy experimental design, editorialism masked as science, data mysteriously disappearing, biases galore, et cetera.
Points two and higher: click to one of these two reviews: Yet Another Study ‘Proves’ Liberal, Conservative Brain Differences, Brain Atrophy Responsible For Religious Belief?
I’ve done many more, but these capture the gist. Strange that these “studies” have a sort of theme to them, no?
Wired had the sense to ask why so many bad studies? Reason one: studies are too expensive. But since scientists must publish lest they perish, reason two: “[T]he pressure on scientists to publish often, preferably in high-profile journals, to advance their careers and win funding from the government.”
Since that pressure will not be lifted even after Button’s identification of systematic flaws, it is rational to expect a continuation of systematic flaws. Gives me a kind of job security, though.
IDing poor science doesn’t pay as well as generating it, however,. Actually it pays not at all. That’s why I think warm thoughts about rum: to keep my brain lit up.
Thanks to Mike Flynn for pointing us to this fine news.