Poor Michael Boren lay dead at one of the ugliest places in New York City, the on-ramp to the Queensboro Bridge. Heart attack. He was only 51. He made it through just two boroughs of Sunday’s Five Boro Bike Tour before ceasing to be (materially).
Thing is, Boren’s doctor “gave him the OK to participate in the race.”
Another busted forecast.
Or rather, another plastered prognosis. Turns out doctors, like most of us, aren’t such great predictors. Human behavior is too complex for anybody to nail with anything approaching consistent accuracy. This includes experts prognosticating in their fields of expertise, too. That was the lesson of, among many others, Phil Tetlock’s Expert Political Judgment.
I was reminded of this when reading a physician’s lamentation of over-confidence, in which he pointed to a British Medical Journal paper “Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study.”
This is just one example of an endless supply. The study: “343 doctors provided survival estimates for 468 terminally ill patients at the time of hospice referral.” There are a lot of words in the paper, but it all comes down to this picture, which isn’t as good as it could be:
The chart is backwards to custom, which would place the predicted survival days on the “x” or horizontal axis, and the observed data on the “y” or vertical axis. The chart is also on a log-log scale, which makes it difficult to appreciate the magnitude of the errors.
But, forgiving all that, let’s take a look. If the doctors gave perfect forecasts, all the dots would line up on the solid black diagonal line: the distance from this line is the error. Not too many dots on or near the line.
Put your finger in the leftmost dot at 30 days, which is one doctor’s prediction of how long his patient would survive. Drop down from that 30 to the x-axis to learn that the patient actually lived just 2 days. That’s a huge error, especially considering this is the End Of The Road, a time when families are making hard decisions.
Because of the log-log scale the errors are larger than you would think in some places. For example, look at topmost dot at just over 1000 days, which is about three years. The patient only lived a month (30 days). That’s a bigger error than the one at the far left, where the doctor said the patient would live around 400 days, but where the patient made it only to the next day (oops). The error is smaller here even though the distance to the black line is longer, because the scale is not linear.
Notice that most of the dots are on the north side of the line which means, for this group of patients and doctors, the forecasts were too much on the optimistic side; that is, the doctors said patients would live a lot longer than they actually did. You can also see a bit of cultural bias in the data: e.g., the cluster of points predicting 90 days (3 months) to live.
One problem in this study is the discrete nature of the prognoses. No doctor and no patient believes he will live precisely 90 days when given that forecast. There is some plus-or-minus which is understood, but maybe not in the same way by both parties. The doc’s window may be narrower than the patient’s, or vice versa.
Every good forecast provides an indication of its uncertainty. A prediction of “90 days plus or minus two months” is different from one which says “90 days to a year.” And of course, doctors more often give predictions in this form. The uncertainty is needed because the decisions a patient and his family makes given a forecast are vastly different than the decisions the doctor makes.
Incidentally, assessing the quality of predictions which come with uncertainty is more difficult than making simple plots like this, but the methods to do so are well understood.
And there’s more to think of. Should a physician give his patients hope by telling them they’ll live longer than he really thinks? “Buck up Mr Jones! I’ve known patients in your condition who lived for years.” Optimism is a sort of placebo, is it not? But can you tell a patient he will live “years” when you believe that patient is circling the drain? Optimism has limits, and the power of the mind (placebo effect) not omnipotent. Bad forecasts aren’t helpful to families, either.
Medicine is an inexact science.
Maybe the doctors are basing their estimates on the wrong disease.
“About 40% of the time in hospital deaths in America, across the board, the clinical doctor gets the cause of death flagrantly wrong, or there is a serious mistake in what is listed as the cause of death,” says Dr. George Lundberg, former editor of the Journal of the American Medical Association.
I always heed my professor’s advice to be skeptical of anything I see presented on a log-log plot.
To err is human, to really screw things up takes a computer.
Be skeptical by all means, but log-log plots are often essential and appropriate. In this case it is needed to show the full range of the data in a clear fashion. A problem would occur if this scatter was fitted to a straight line and one attempted to find significance in the slope. The line drawn just shows the one to one ideal and indicates the imbalance of prediction.
To Briggs et al.,
The prediction imbalance occurs because we the patients, or the family of patients, want an answer to an extremely difficult question that the physicians would rather not address. The only meaningful answer in all these cases is soon. It reminds me of the rope skipping rhyme. I’m sure that you know the one. You seem to be taking a macabre turn recently. It is very depressing.
Doctors are being paid to *cure* people. So the thing to worry about is how good they are at curing people. And the prediction to check is how good they are at predicting that a treatment will actually cure the patient.
See 10:29 for Dr. Rahmat’s answer to Rumpole’s question, “Are you a brilliant doctor?” http://www.youtube.com/watch?v=8yXy6-dOnNI
I recall reading, somewhere, long ago, that the best diagnosticians were “good guessers”. Even at that time, the article noted that every decision down to determining which test(s) to run required a certain amount of luck along with experience and education, and those doctors most willing to follow their hunches had as good or better a result as those who operated by a checklist of elimination. It sounds as though this may still be the case, despite the proliferation of diagnostic tools today.
Interesting although I had to ask what a log log scale was. Part of the problem with prediction of mortality is that doctors need a certain amount of intuitive reasoning. Since most MD’s now are trained as engineers, that is a problem. Engineers would score about zero on an intuitive test. Second, predictions of mortality tend to rob the patient of hope and accelerate their demise. Finally, nobody is born with an expiration date. It is foolish to pretend that we do. Medicine is practice. Looking only at the statistics is way too narrow.
I remember a paper by another statistician on how upset he was to receive a diagnosis of “three months to live” only to think it through and be much less upset. His thesis went as follows:
Most of the people diagnosed with his disease were diagnosed at death.
The doctor based his estimate on previous information.
Thus, if the the three months was an estimate of time from diagnosis to death, he could expect to live longer than three months, given that he was diagnosed while still alive. Furthermore, all of the people who died of his disease died in the past and with advances in treatments, he could reasonably expect to live longer the previously diagnosed patients.
By the way, I believe the paper was written some five years after the diagnosis was received.