Statistician to the Stars!
Background
You often hear, if you manage to stay awake during the lectures, a mathematician or physicist say, “The following is a necessary but not sufficient condition for my theory to be true.”
We say this so often that we tend to blend the words together: necessarybutnotsufficient, and we forget that it can be a confusing concept.
It means that there is an item or a list of items that must be the case in order for my theory to be true. But just because that item or those items on that list are true, it does not mean that my theory must be true. It could be the case that the item is true but my theory is false.
This is all important in the theory/model building that goes on in the sciences.
For example, it is necessary but not sufficient that my theory be able to explain already observed data that I have collected. If I cannot at least explain that data, then my theory cannot be true. This is necessarybutnotsufficient in the weak sense: all theories must be able to explain their already-observed data.
Again, it is a necessary but not sufficient condition that my theory be able to explain future data. This is necessarybutnotsufficient in the strong sense: if my theory is right, it must be able to explain data that is not yet seen.
Understand: it can still be the case that my theory can predict data that is not yet seen and my theory could be false. This is true in all cases where we cannot deduce (know with certainty) the true of a theory. Most theories (outside math) are not, of course, deduced.
How about an example? Let’s us the following game.
The Game
Go to The Philosopher’s Mag and play this game called “Dealing with Induction”. It asks the question: “how easy is it to draw a wrong conclusion about the future from the evidence of the past?”
The game tests your inductive reasoning skills and asks you to infer the rule that accepts or rejects cards from a standard 52-card deck.
Let me be as clear as possible. The following conditions hold: (1) You believe that a rule that generates the card exists. (2) You will see a sequence of cards from which you will attempt to infer, through induction, the rule. (3) No matter how many cards are shown you will never know the rule with certainty; that is, you will never be able to deduce the rule from a set of premises.
Do not read further until you have played the game fully and discovered its secret.
Did you really play?
Tell the truth. Don’t cheat and read any more until you have played the game.
I have been asked to write this review by a party who wishes to remain anonymous because of the fear of reprisal. See this article for background. Also this.
Dahn yoga might not be of interest to you, but this review is larger than that. It will show you how easy it is to publish material in a well known journal that is poor at best. Civilians are often shocked to discover that peer-review is only a weak indicator of correctness. This article removes some of that mystique.
This article is long, but has to be.
Study Abstract
Sung Lee, an internist, while at the Weill Cornell Medical School, conducted an experiment of Dahn yoga. He published the results in the paper: Prospective Study of New Participants in a Community-based Mind-body Training Program. It appeared in the Journal of General Internal Medicine, 2004 July; 19(7): 760–765.
This study has been touted by Dahn sympathizers: Ilchi Lee (Dahn founder) boasts of it (pdf), TV broadcasts (Sung Lee’s picture is in the upper-right corner), as supportive research, chiropractors in Sedona, and fan sites of Dahn.
Lee went to several locations in New York City and recruited people to join a three-month Dahn (introductory) program. All of these people self-selected themselves into the program; they came seeking yoga; all were new to Dahn. All were asked, at the beginning and end of the study, a series of questions.
The main ones were from the SF-36, a standard questionnaire. The most useful SF-36 “item” is question 1: “In general, would you say your health is: Excellent (5)…Poor (1)?”1 Another is, “Have you been a very nervous person? (6) – (1)” This “instrument” is divided into domains, such as “vitality” and “mental health,” which are simple functions of the questions. The “nervous” question is part of the mental health domain. Another is, “Have you felt downhearted and blue? (6) – (1)”
Papers which use the SF-36 rarely show the questions; they are content to report on the domains. Seeing the exact questions makes them sound far less impressive than when stated in their usual academese. For example, Lee calls the SF-36 “a validated health assessment instrument.”
Averages of the US population of each of the domains exist. The participants in the Dahn study began with scores lower than the US average: a fact which is not surprising, considering these were people who were newly arrived for exercise training.
There was no control group: all received the Dahn training. 194 started, and 171 completed the study. Three out of four were women. Five of the 171 reported an injury due to the training: it is unknown how many of the twenty-three who dropped out were injured.
From the abstract: “New participants in a community-based mind-body training program reported poor health-related quality of life at baseline and moderate improvements after 3 months of practice.” This means that several of the people had small increases in their three-month SF-36 scores.
That is, some people went from answering “A good bit of the time” to “A little of the time” on the question “Have you been a very nervous person?” And so on for some of the other questions.
From this, Lee was able to say that “Dahn worked.” Actually, the best that could be said was “Dahn didn’t cause too much harm.” Here’s why.
Specific objections
I was at Cornell at the time Lee was completing, presenting, and writing up his study. I made my objections known at that time. You must also understand that in academics “A paper is a paper”, and nearly anything can be published in some peer-reviewed journal somewhere. Because of this, the number of journals is staggering: they increase constantly.
