Julian Champkin, editor of Significance magazine somehow came across the percipient insights of yours truly and asked me to write l’article controversé. Which I did. And with gusto. Champkin, a perspicacious individual with the insight and experience of one long accustomed to the peculiarities and peccadillos of publishing, added the word “silly” to the title. I find myself not objecting.
The article is here. I beg forgiveness that reading the piece requires a subscription (yours or an institution’s).
Statisticians are, in actual fact, as an Englishman would put it, Significance being an organ of the Royal Statistical Society, in the bad habit of answering questions in which nobody has the slightest interest. More rottenness is put forth in the name of Science because of the twisted cogitations of statisticians than because of any other cause.
The problem is that the questions statisticians answer are not the questions civilians put to us. But the poor trusting saps who come to us, on seeing the diplomas on our walls and upon viewing the perplexing mathematics in which we couch our responses, go away intimidated and convinced that what we have told them are the answers to their queries. They can’t, then, be blamed for writing results as if they had received the One Final Word.
There are many reasons why we lead our flocks astray, but the main culprit is we instill a sort of scientific cockiness. A civilian appears and asks, “How much more likely is drug B than drug A at curing this disease?” We do not answer this. We instead tell him which drug, in the opinion of our theory, is “better”, imputing a certainty to our pronouncement which is unwarranted.
We’re tired of these examples, but they are paradigmatic. It is through the wiles of statistics that sociologists can “conclude” that those who either watch a 4th of July parade or who see, oh so briefly, a miniature picture of the American flag can turn one into a Republican.
The old ways of statistics allowed over-certainty in the face of small samples sizes. The new ways of doing statistics (now not always called statistics, but perhaps artificial intelligence, data mining, and machine learning) allows over-the-top surety in the face of large sample sizes, a.k.a. Big Data. The difference being that the later methods are automated, while the former are hands-on. False beliefs can now be generated at a much faster rate, so some progress is being made.
If you followed last week’s “Let’s try this again” on temperatures, you’ll have an idea what I mean about over-certainty (incidentally, due to time constraints, I will not be able to answer questions posted there until tomorrow). Also click the Start Here tab at the top of this page and look under the various articles under Statistics.
Update Posting date change to allow more comments.