The Unseen Bias In All Science

I picked a lousy (slow, pre-Fourth) day to post a criticism of “IQ” last week, and badly titled it “Test Your IQ With These Puzzles! (Not So Easy!)“. Something like “A Hidden Bias in IQ Tests” would have been better, perhaps. That, and asking people to have a go at puzzles can be intimidating, or perhaps seem pointless. Anyway, too clever by half, as they say.

(Which reminds me, follow me on my YouTube channel. Do this as quickly as you can.)

Briefly, all IQ test questions have a “bias”, where I use that word in an entirely neutral way. You cannot remove conditions, i.e. “bias”, from questions, just as you cannot remove tacit premises in any logical proposition that always include the definitions and meanings of the words and symbols used. That being necessarily so, the IQ test itself cannot be a perfect measure of intelligence, and must be in part, however small, also a test of bias recall. (This adds to my other criticism of attempts to quantify intelligence.)

This isn’t a devastating blow “against” IQ tests, but it is a good point, and it’s well to keep in mind when, as some do, thinking about differences in group scores.

Even though my post died the death, one answer I did get was a gentleman insinuating my critique was wrong because he said IQ tests are the best measures we have of intelligence. Accepting that IQ tests are the best ways of assessing intelligence, at least for the sake of argument, it does not follow in any way, not even by a scintilla, that therefore there is no such thing as inherent and irremovable bias in questions. This would be like a caveman arguing that since sharp sticks are the best known way to hunt animals, all demonstrations showing they will not penetrate the hides of some animals are invalid.

I grant IQ tests are the best way, to date, of assessing a certain kind of inherent analytical abilities. And I have said innumerable times, and say once more, that IQ tests are useful and helpful, and that the many stereotypes we hold about intelligence are true. But I also insist that “IQ” is not intelligence: intelligence is intelligence, and that if you mean intelligence and not some score on a biased, but useful, test, then say intelligence and do not say “IQ”.

That off my chest, the best criticism I had was from my dad. I gave him the first question on the quiz (What is the next number in this sequence: 1, 2, 3, 4, 5, ___?) After I gave him the answer (which isn’t 6) and its explanation, he asked, politely and calmly, “What good does any of this do?”

One, and a minor one, to critique a specific aspect of IQ test. But mainly to help us notice there is much over-certainty in science.

Quoting the speech I gave a Hillsdale (from an even greater number of similar examples; also at YouTube):

Ioannidis examined forty nine top papers. Here’s what he found: “…7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged.”

…

The British Medical Journal 2017 review of New & Improved cancer drugs found that for only about 35% of new drugs was there an important effect, and that “The magnitude of the benefit on overall survival ranged from 1.0 to 5.8 months.” That’s it. An average of three months.

Richard Horton, editor of The Lancet, in 2015 announced that half of science is wrong. He said: “The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.”

I’ve warned us many times about new drugs. These do well in the hands of drug companies. Companies duly report their mandatory statistics, using approved and customary methods of analysis, and then the drugs are made official and released into the wild. At which time their initial successes diminish. Or even disappear. Or become failures.

Leaving aside all questions of cheating, which of course is with us always, and even passing by all questions of analysis methods, of which I am hyper-critical, a reason for this typical diminishment of signal is unacknowledged and unseen biases.

One way to say this is that in these cases we bring cause to data, and data do not bring cause to us. We begin in all these analyses by assuming the new drug (or whatever, as this applies to all science) is causative in the direction we desire. We next make a guess of how strong the cause is, which brings in yet more assumptions. These are both biases, which don’t seem to be there, think many, because of the bizarre way the analyses proceed (which gives us wee Ps and all that).

The example I like to give is that we do not include the color of the physicians’ socks who administered the new drugs in the experiments in the models of the drugs’ efficacies. Because why? Because of our bias—again, using the word in it neutral meaning—against sock color being causative.

Bias is not bad! Nor good. Not inherently. It only becomes good or bad when the bias we assume does or does not match Reality. Consider not only do we leave sock color out of these models, we leave out an infinity of possible other causes! Hence the real point of last week’s post. Most of these omissions will be correct, as most things we can imagine (say, aliens in the Andromeda galaxy shooting G-rays at us) won’t be causative in our experiment.

I don’t only mean scientific experiments, but all cases in life where we are assessing what caused whatever we take an interest in.

If you’re doing an experiment testing a new drug, it is both natural and necessary to suppose it is a potential cause in whatever effect you are measuring. This is a good bias. The problem, the bad bias, comes in not supposing other causes which may be the only ones, or the main ones, operating in your experiment. And that is even natural, too, since many times these other causes won’t be measured well because the experiment is not about these, but about the drug.

And because you are so used to leaving out potential causes—after all, there are an infinity of them—that it’s really nothing to leave out a few more. Then comes the weeping and gnashing of teeth.

Not noticing what’s implicit or tacit in questions, whether in IQ tests or science experiments, is common. Trying to tease out what you don’t see is hard work; unseen good biases can elude even the best minds. Consider the very best experiments (always on things and not people) in science are those that try to control every possible cause, and manipulate only one of these, so that any effect can be ascribed to the one cause that was manipulated. But even these still leave out an infinity of other potential causes, and so it is an assumption, a bias, that you have found the one single correct one.

The main point being that this is inescapable. It is not a weakness that can be eradicated. It is part of our nature, because we do not have the ability, and never will, to see all things.

Subscribe or donate to support this site and its wholly independent host using credit card click here. Or use the paid subscription at Substack. Cash App: \$WilliamMBriggs. For Zelle, use my email: matt@wmbriggs.com, and please include yours so I know who to thank. BUY ME A COFFEE.