Thanks to the Wrath of Gnon, we have this report: Sorry, wrong number: Statistical benchmark comes under fire, from Phys.org.
Earlier this fall Dr. Scott Solomon presented the results of a huge heart drug study to an audience of fellow cardiologists in Paris.
The results Solomon was describing looked promising: Patients who took the medication had a lower rate of hospitalization and death than patients on a different drug.
Then he showed his audience another number.
“There were some gasps, or ‘Ooohs,'” Solomon, of Harvard’s Brigham and Women’s Hospital, recalled recently. “A lot of people were disappointed.”
One investment analyst reacted by reducing his forecast for peak sales of the drug—by $1 billion.
The number that caused the gasps was 0.059. The audience was looking for something under 0.05.
I’ve said it precisely three thousand four hundred and seventy seven times, give or take a million: p-values are magic. They encourage magical thinking. Science results based on p-values become so much mumbo jumbo.
Get a 0.049 and people start doing Harry Potter imitations, zapping spreadsheets with their magic wand fingers. Which would be only just tolerable if it weren’t for the manic giggling that accompanies these dreadful performances. Get a 0.051 and they start sharpening their seppuku blades. Tears ain’t in it: the lamentations of grown men cursed by the Wee P gods is a pitiable sight. You’d sooner gaze upon Cambodian dogs in a cage being readied for the fryer.
Something must be done. And I know what that something is.
Now I have a whole series of papers, linked here, which contain a dozen or two knock-out death-blow killer guts-ripped-out-and-roasted-over-a-spit-until-charred arguments proving the illogicalities of p-values, proving they should never be used, proving that every practical use of them involves a fallacy.
I showed these to many people. From Judea Pearl, no p-value fan, I got a blank stare. From Deborah Mayo, the palsgravess of p-values, I got the cold shoulder. From many others I got “Who’s this Briggs guy?” “Nobody. Ignore him.”
Now this is only right and natural. I do not complain. Approvingly quoting a guy like me can make a person take on my characteristics. Not the handsome courageous strapping, manly 6’2″ 200 pounds of steal characteristics. No. What they end up with is the smear of associating with a racist sexist homophobic transphobic anti-Semitic Islamophobic xenophobic abelist. Once you’re accused of even one of these, clearing yourself and proving your social justice bona fides is like trying to rid yourself of super-glued double-sided sticky tape one-handed.
There are some academics out there, some even quoted in that piece, who know of what I speak. But I dursn’t out them, because I don’t want to get them into any trouble, and because what I want is not additional fame—I reside already at the pinnacle, unbudgeable—what I want is for people to stop using p-values.
So we’re going to have to go about this in another way.
I suggest plagiarisation. Take the arguments in those papers and claim them as your own. You will not hear any whining from me. Use them and show the world how it errs. Bruit them around at academic conferences. Say “Here’s an argument that proves what you’re doing is wrong.” Be haughty. Take credit. I will celebrate your success and shake your hand, and, if you catch me on the right day, I might even buy you a beer.
Why such awe-inspiring laudable praiseworthy award-eligible generosity from such a humble person like myself?
Because even after you whack p-values, you’re still screwed.
You can swap p-values for Bayes factors, if you have a mind, or even confidence or credible intervals. But you will still be talking about fictions. You’ll still be focused on unobservable non-existent parameters inside some ad hoc model. You won’t be right about cause, except by accident, and you won’t be right about the uncertainty in the observable. You’ll still be wrong.
By which I mean, you’ll be right about the wrong thing.
Why not instead give people what they ask for. Some poor sap goes to a statistician and says, “What happens to the Y if I change this X?” And that statistician says, “I don’t know. But what I can tell you is that if repeated your experiment an infinite number of times, exactly the same each time, except each is randomly different, where nobody knows what random means, then the chance of you seeing a test statistic larger (in absolute value) than the one you actually got is 0.051. Please don’t kill yourself.”
Whereas a statistician focused on Reality would say, “There’s a 10% chance of Y.”
Switch to the predictive way.