Today’s guest post is in the form of an extended email by Gerald E. Quindry, Ph.D., P.E. who noticed something peculiar about some official findings on methane measurements.
Dr Briggs,
In many of your posts, you have commented on the use, and abuse, of statistics in publications. Here is an example on which you may wish to comment. The documents I discuss below describe an investigation into the quantity of methane released from natural gas pipelines on the island of Manhattan. Both the preliminary and extended reports can be accessed from the Internet at: PDF link.
I came across these reports while searching for better, inexpensive method to monitor methane gas concentrations in air. I am an environmental engineer, and I have designed methane mitigation and monitoring systems for buildings constructed in areas where methane gas intrusion is a potential problem in building design. It may surprise you to learn that construction in a considerable portion of my home area of Southern California faces this issue. If you’ve ever been a tourist in Los Angeles, and visited the La Brea Tar Pits, the gas bubbles you see in the pits are…methane!
What first set off my “BS meter” was the following, from Page 20 of the “Extended” report:
These data were as follows:
Methane Concentrations in Ground-Level Air
Upwind 1.92 ppm ±0.003 ppm (99.9999% Confidence Interval)
Downwind 2.165 ppm ±0.021 ppm (99.9999% Confidence Interval)
I can’t recall ever before seeing chemical data presented with a “six nines” confidence interval; certainly never from data containing the kind of spatial and temporal variability that would be present in these data. I leave it to you, the “Statistician to the Stars” to explain how the numbers can be misinterpreted.
My real heartburn with the studies was the subsequent use of these data. These concentration values were used to calculate the change in methane levels in the air as it moved across Manhattan. Subsequently, that change was used to estimate the amount of methane released from the natural gas distribution piping underlying the City. (Natural gas is 85-95 percent methane.)
But a number can be both very precise, but very inaccurate, at the same time?
There are many potential sources of error in the analysis performed by the study. For example, there are many other sources of methane in the city, and no accounting of these other sources was made. Automobile exhaust contains methane from incomplete combustion of fuel. Sewers emit methane from the anaerobic bacteria that flourish there. Landfills and trash piles emit methane. Humans and animals emit methane. I also am doubtful that the analytical method used in the study was specific only to methane.
Many other volatile chemicals are released into the atmosphere, at restaurants, gas stations, dry cleaners, and the like. Paint contains volatile organic chemicals that are released while the paint dries. It is unclear that the data collected during the study would not be contaminated by passing near a release point for one of these air contaminants. Finally, (and this gets even deeper into the report methodology) it is my opinion that their treatment of boundary conditions and vertical mixing introduces huge potential errors in estimating the quantity of methane released to the atmosphere.
In summary, driving around the City, taking thousands of measurements of gas concentrations at the surface of the congested city streets, and then doing the calculations presented in the paper seems to be a poor choice in an attempt to quantify methane leaks from buried pipes.
The question then arises, why do the study? In my opinion, it is simply a side-battle in the climate change propaganda war.
Natural gas production in the United States is rising dramatically, due to advancements in the technology for drilling and extraction. The resulting oversupply of gas has dramatically reduced the price. That, in turn, has made it much more difficult for alternative energy projects to be economically viable without large, continuing, and reliable government subsidies. That has generated the need to discredit natural gas from its status as an economic, clean, and reliable energy source.
To accomplish this, the reports are publicized in a press release, “Natural Gas Emissions Measured in Manhattan Showing No Advantage to Natural Gas: Two Reports” (PDF) which is then picked up by the press and discussed on activist web sites, such as “New Study Exposes How Natural Gas Isn’t the Clean Fossil Fuel It’s Hyped up to Be” and “No smell of gas – but is that really OK?”
In my opinion, this shouting from the extremes, practiced by both sides in the debate, is no way to develop the sound energy and environmental policy that we desperately need.
——————————————————————————-
Back to Briggs
Skipping the bizarre frequentist interpretation of confidence intervals and instead thinking like Bayesians, the “Upwind 1.92 ppm ±0.003 ppm (99.9999% Confidence Interval)” means that there is only a one in a million chance of seeing an upwind methane value outside the bounds 1.897 to 1.923 ppm for air coming into Manhattan. That’s strange because on p. 4 of the original report “Open country” values of 1.787 and 2.484 ppm were found.
They needed that narrow interval for their theory, though, which shows how much methane was supposedly added to air as it passed over Manhattan. If the interval on the incoming air was large, as it apparently should be, they could not claim leaky pipes added “8.6 billion cubic feet per year” (with no plus or minus) of methane.
Strangely, the confidence they claim on p. 4 is even higher: “highly reliable, 99.9999% confidence intervals ± <1% (0.021 ppm).” That’s sure some kind of confidence, boy.
And yes, a number can be both very precise, but very inaccurate, at the same time. You could have a gauge which prints readouts to arbitrary precision but which isn’t calibrated (the gauge reads “6.48572” but the actual value is “42”).
But skip all that. The most important part of Quindry’s criticisms are the other sources of over-confidence which the report does not address (such as the car exhaust, etc.).
It appears to be from the grey literature, published by an advocacy group, and not a peer reviewed journal. Small comfort, maybe, but it explains the complete disdain for rigor. Or maybe not.
William S,
That appears right. But that doesn’t mean government wouldn’t rely on it. Wouldn’t be the first time.
A high confidence level is usually associated with a wider interval. For example, X±∞ is a 100% confidence interval. I am absolutely certain the mean value lies in that interval.
I’m pretty sure the confidence interval is for the mean value and not for individual measurements. Those would be “prediction intervals.” So it would not be unusual to find measured values outside the confidence interval.
What I would first question is whether there is a mean value, in the sense of a “central tendency.” Like the old joke runs, if you stick your head in the oven and your feet in the freezer, on the average you’re comfortable.
YOS,
That’s right too, though the difficulty is that decision makers don’t know the difference. And some of those decision makers are the compilers of the statistics. The certainty in the parameters is all too often taken to be the uncertainty in the observable.
Plus (for other readers) prediction intervals are what is wanted here, not parameter intervals. Who cares about a parameter? We want to know about the methane!
Whoever wrot up the report evidently cares about the mean parameter. What kind of prediction intervals do you have in mind? Why kind of policy decisions can be based on them?
While you make no attempt to provide any evidence for your OVERCONFIDENT assertions such as the following, (Perhaps you have some metaphysical powers.) yet you constantly criticize how others try to do their best to devise various instruments to collect data and provide some evidence, albeit not perfect.
Why? If you cannot provide answers to my questions that I want, I promise I won’t curse you as Jesus did to the fig tree.
(Too lazy to edit my comments.)
JH,
Dude, you’re mixing statistical metaphors, as it were.
Of course we do not care about parameters when the intention is to predict how much methane is added to air as it wafts over Manhattan. Parameters depend on models, which are empirical theories, the only proof of which is that they make good predictions. We could posit any number of models whose parameters say anything (implicitly) about methane. But what’s wanted is the real measure of uncertainty of real methane.
We’ll the other matters to the relevant thread.
For example, the calculation of a mean is already a model and implicitly assumes that there is a central tendency and that there is a statistical distribution about this mean. In quality control work, we often find that there is no statistical distribution, but rather a heap of many possible distributions, and the mean value may change from place to place or from time to time.
There is a discussion of the difference between confidence intervals and prediction intervals here: http://tofspot.blogspot.com/2012/04/lets-be-precise.html#more
JH,
Here are some more considerations regarding overconfidence and bias in the publications I reviewed.
In calculating the mass of methane entering the Manhattan air space, and then doing the same calculation for the mass of methane leaving the Manhattan air space, several assumptions need to be made. One is that the concentration estimated by the sampling effort is representative of the entire volume of air entering (or leaving)the air space. But samples were only collected near the ground. One can decide on any one of many dispersion models. The most extreme cases are:
1. No vertical mixing, and
2. Complete, instantaneous dispersion through the entire 2,600-foot vertical column of air.
Reality is somewhere in between these two extremes. The authors chose Number 2, rather than a practical middle-ground. They assumed complete mixing. That will overestimate the mass of methane released in their example, because their estimate of downstream mass will be high, due to incomplete mixing.
On the other hand, in attempting to get the most impressive confidence interval, they did many, many readings of their instrument as they drove. Are these samples independent? The gas sample enters a tube near the pavement; presumably it then passes through a filter, then more tubing, and maybe a pump before entering the chamber where the reading is taken. (The pump may be after the chamber, and then would not apply.) During that transit, the gas collected at one instant is blended with gas collected before and after. The gas was analyzed very frequently to allow many samples to be collected. I very much doubt that they could be considered independent, unless no mixing is assumed. But completely opposite to the situation with mixing of the vertical air column, the authors chose to assume no mixing here. This will lead to overconfidence in the result.
Mr. Briggs,
Dude, it’s true that different models have different parametric setups. The predictions will also depend on the modes and the parameters. Parameters are not observable; therefore, it’s impossible to evaluate the goodness of the uncertainty assessment about them, which doesn’t mean they can’t be interpreted or have no physical meaning.
For linear models (a constant model such as the one used here is also a linear model), and sometimes nonlinear, the parameters have direct interpretations. You know how to interpret them; see your own post here.
FDA might want to know if drug A is better than drug B ON AVERAGE (parameter) ACROSS MANY PATIENTS when approving drug. A doctor might want to know how much more will drug A lower A SPECIFIC PATIENT’S blood pressure (prediction) given the patient’s characteristics.
One would want to estimate/assess the uncertainty based on information/data available as best as possible. No true/real measure of anything. Perhaps you really have the illusion that so-called objective Bayesian (see
here and
here.
) can give you real measure of uncertainty!
Mr. Gerald Quindry,
You are probably right. I usually try not to read studies such as this because I would want to figure out whether there is or how I can postulate a model to accommodate all possible errors and the data structure, be it possible or not. Which is hard to do because to do so, a statistician also relies on the subject area experts.
±0.003 ppm
I am very inpressed. They can measure 3 parts per billion tolerance. When I worked on the AF missile range at Cape Canaveral we had a metrology lab that calibrated all our test equipment. They had all kinds of standards and measureing equipment and they couldn’t measure anything in parts per billion. I was just wonder what metrology lab calibrates their methane measuring instruments. The rule of thumb in our lab was that the instruments used for calibration had to be ten times more accurate than the equipment you were calibrating.
±0.003 ppm. I am very inpressed. They can measure 3 parts per billion tolerance.
To be fair, ASTM rules allow an extra decimal point in the calculation of means and the confidence interval is a bound on the mean. That is, if you have numbers like 1,2,5,7,8 the mean can be written as 4.6 even though none of the measurements are good to a 0.1. If you have hundreds of X’s, X-bar can have two extra decimals.