Statistics

CARB Misinterprets Statistics, Calls For Elimination Of Dust

The California Air Resources Board, or CARB, has issued a press release that shows how easy it is to misunderstand statistics. That most do—fail to comprehend what statistical results are and mean1—is no surprise, but its particularly disheartening in CARB’s case.

Why? Because its misconstrual is followed by a call for increasing the size and scope of government. This exercise once again shows how willing bureaucracies are to latch on to any study that might, however tenuously, be in their favor. It is therefore a useful, if achingly dull, exercise to examine the press release and its history. CARB

It begins with: “Fine particle pollution a threat to the cardiovascular health of Californians.” (Fine particle pollution is, among other things, dust.)

Three new studies released today by the California Air Resources Board reveal that exposure to airborne fine-particulate matter significantly elevates the risk for premature deaths from heart disease in older adults and elevates incidence of strokes among post-menopausal women. Heart disease is the number one killer in California and is responsible for approximately 35% of annual deaths.

Before we get to the meat, note that there will always be a “number one killer” so your antenna should always be up when somebody begins discussing what it happens to be this year. Secondly, notice the specificity of the claims. CARB isn’t stating that all citizens will develop heart disease or stroke after exposure to “airborne fine-particulate matter”, but that only some old people and post-menopausal women will (this latter group presumably will suffer both strokes and heart attacks). Specificity does not imply falsity of claims, but it is often the case the more specific the medical claim the more the suspicion that data snooping (searching for publishable p-values) has taken place.

That indeed is the case for one of the studies quoted by the press release: the statistical research of Michael Jerrett et alia. I reviewed in depth Jerrett’s study here and here. Briefly, here is what Jerrett did:

He gathered data on the nearness to possible sources of PM2.5 (the fine particulate matter) where Californians possibly lived at one point in their lives, then he noted whether the person died and of what by the study’s close. His original goal was to demonstrate that the greater PM2.5 exposure lead to higher death rates. This goal failed, in the sense that his statistical researches did not generate enough evidence to say so.

But the data he collected allowed him to test multiple causes of death, so he searched until he found one cause among the many. This cause produced publishable p-values. He never adjusted for the multiple testing, which always gives p-values smaller than they should be. But never mind that. And forget that the sources of pollution and cardiac death rates differed by an order of magnitude between urban and rural populations and that the model Jerrett used did not adequately account for this, therefore improperly inflating his confidence.

Ignore these and all the other mistakes and focus on the conclusion, which stated that “exposure to fine particulate matter significantly elevated the risks for premature death from heart disease.” But this is not what the kind of statistical models Jerrett used can say. He never measured the actual PM2.5 exposure of even one person. There is no causality, only suspicious correlation in his models. And these models say nothing directly about even the correlation: they instead make statements about functions of certain parameters in the model.

The press release should have read: “Small unadjusted p-values were found in one of several models that say a certain parameter associated with once possibly living near a source of dust might be correlated with a fractional increase in the possibility of heart disease.” A mouthful, but true and justifiable, unlike the actual headline which possess the opposite of these traits.

Incidentally, I and several others gave extensive critiques to CARB, which accepted and discussed them at an open meeting. My comments were considered, but ultimately dismissed by the CARB panel which argued that the errors I identified were of the sort common in studies like Jerrett’s, and that since these other error-ridden studies had been accepted, so should Jerrett’s.

This fallacy is indefeasible. It is also rife, the essence of bureaucratic scientism. It is the grown-up’s version of the teenager’s everybody-else-is-doing-it argument. The if-everybody-else-jumped-off–cliff retort is nevermore heard.

The press release kicker: CARB Chairman Mary D. Nichols reasoned that “These new studies underscore the need to eliminate the threat [of dust] from California’s air.” The only threat that has been demonstrated is Nichols’s anxiousness and determination to burden us with more rules and regulations.

—————————————————————————-

1The fault lies with we statisticians, and begins in the classroom.

Categories: Statistics

12 replies »

  1. If you can’t teach’m (“beat’m”) then join’m (push them along their self-chosen fantasy path to some ridiculous conclusion that makes their idiocy apparent to all).

    Here, wholly endorse CARB’s efforts at reducing fine particulate matter (“dust”) and note that since much of that is imported from China that what’s clearly necessary is a team of diplomats to visit China to persuade them to get the Gobi Desert to expatriating their dust into California without permission. Local littering ordnances should suffice for collecting due payments….and since China is so flush with US cash they’ll have not problem paying.

    Also, as not only is much of California’s dust from China/Gobi — that same dust affects health, precipitation (an estimated reservoir of water each year from precipitation lost by weather-induced effects), and, climate is [of course] disrupted adversely [is there any other way?]:

    http://www.inthesetimes.com/article/2849/ : “A new Chinese export has been spreading quietly across Asia and the United States: dust. … Violent sandstorms from China’s expanding deserts have been battering numerous Chinese cities, and now their mustard-colored dust has begun reaching South Korea, Japan and the west coast of North America. … “People dusting off their cars in California or Calgary often don’t realize the sand has come all the way from China,” says Lester Brown, president of the Earth Policy Institute (EPI) in Washington D.C., who was in Beijing recently. “There is a dustbowl developing in China that represents the largest conversion of productive land to desert of any place in the world … and it’s affecting the world.””

    AND: http://discovermagazine.com/2005/mar/cover

    http://www.paloaltoonline.com/news/show_story.php?id=11731&e=y

    http://articles.sfgate.com/2002-03-28/news/17535644_1_dust-storm-air-pollution-sandstorm

    http://www.usatoday.com/weather/news/2001/2001-04-18-asiandust.htm

  2. There is nothing new about bureaucrats faking the statistics. It’s commonly called lieing. However, as one of the people who worked on the EPA report on second hand smoke said, it’s rotten science but it’s for a good cause. Lieing for a good cause is ok. It’ like Dan Rather’s fake but accurate memos. No need fact check because it was for a good cause.

  3. Dang it! Taking the fun out of living in a dust bowl. Is nothing sacred? Always thought playing in dust was more fun than splashing around in mud. Just goes to show I was wrong again. Should of listened to my wife.

    Does this mean CA will have to “fix” those huge sections of desert freeways prone to seasonal dust storms? Maybe by enclosing those miles upon miles of multi-lane bliss inside some type of extended tunnel? Wow! Can I bid on that job? Will the CARB fund the project? You say I’m unqualified? Oh, yeah. Wrong gender and race. Sorry, for a moment I forgot.

  4. Waiting for Obamacare to kick in and then the death panels can use dust as a cost conainment device

  5. Matt,
    Do you think the CARB people you presented to actually understood that their approval, if that’s what it was, was of a study which had the same problems as the others which they had approved? From which it might follow that they cannot disapprove this one for those reasons without acquiring some obligation to recall all the other equivalently flawed studies? And what about the regulations based on those known to be flawed studies?

    Are there no conscious citizens out there?

  6. I was going to make a joke about fining people with dandruff then I thought, “This is California where jokes become legislation” so I didn’t.

  7. An excellent, and devastating, critique for a deserving target, well done. This is what good investigative blogging is all about – proving the authorities’ ineptitude with solid logic. They should pay you handsomely for this service.

    When you said “At the least, these criticisms call for additional study before any decisions are
    made regarding PM2:5 inhalation and mortality.”, I might have called for better statistical analysis of existing data before recommending more study. What we do not need is another study from these guys.

Leave a Reply

Your email address will not be published. Required fields are marked *