Reminder: The Thursday Class is only for those interested in studying uncertainty. I don’t expect all want to read these posts. Pease don’t feel like you must. Yet, I have nowhere else to put them. Your support makes this Class possible for those who need it. Thank you. MATH ALERT!
We need to learn how to read scientific papers. They all (except, in our days, those devoted to propaganda) have the same goal of arguing for or against the cause of some thing. But they are often not structured to do that properly, becoming bogged down in meaningless tests. Today a prime example.
Video
Links: YouTube * Twitter – X * Rumble * Bitchute * Class Page * Jaynes Book * Uncertainty
HOMEWORK: Read the paper!
Lecture
Before we return to entropy, I want to cover another modeling exercise that made the news. News—about aliens and UFOs, to be exact—extracted from the peer-reviewed paper “Use of machine learning to enhance detection of transient astronomical phenomena in historical observatory images” by Stephen Bruehl and others in Nature: Scientific Reports.
They hint aliens knew about our Twentieth Century nuke tests, and glowed brightly in recognition on the days of these tests, plus or minus one, glows which revealed themselves on the photographic plates used at the time to take pictures of the sky at the Palomar Observatory. Does this make sense? Why do they say this? How do they say this?
Why, in the same way a legion of such works make claims. Which makes this an ideal study to examine. Which means, as we read this together, the criticisms made, while directed at this paper, are of the sort you ought to automatically make when reading any claim. We’re after critical technique here, and not this particular paper per se.
The Abstract (do read it all):
Transient, star-like point sources that appear and vanish over short timescales are described in astronomical images prior to launch of Sputnik. We have reported that transient numbers diminish significantly in Earth’s shadow (shadow deficit) and are more likely within (+/-) one day of nuclear testing (nuclear window). These findings remain debated; some argue transients identified via existing automated pipelines are simply plate defects. Machine learning (ML) was used to enhance transient identification accuracy and validate the phenomenon. The model was trained against 250 transient image pairs taken 30 min apart classified as real versus plate defect by expert visual review; it demonstrated good discrimination (out-of-fold AUC = 0.81; sensitivity = 0.71, specificity = 0.71). After deployment in a dataset of 107,875 previously-identified transients, the model assigned each a probability of being real. After controlling for ML-identified artifacts, transient counts were significantly elevated for dates within a nuclear window (p = 0.024); transients with the highest probability of being real were more likely to occur within a nuclear window (p<0.0001). The shadow deficit was significant (p<0.001) and largest in the highest probability transients relative to lower probability transients (p=0.003). Results support existence of an unrecognized population of transient objects in historical astronomical plates warranting further study.
Here’s a picture (green circles; hard to see; in the center) of three transients:
And now the end of the paper, in the Discussion.
If one accepts that both key findings in this study represent real phenomena and their interpretation is necessarily limited to hypotheses that can simultaneously account for both findings, we are left with a very small set of provocative hypotheses that would be considered highly implausible by most. For example, is it possible that unknown to the public there were multiple launches of artificial satellites long before Sputnik with some launches timed to coincide with U.S. nuclear tests? Or rather, do the current findings represent detection of a non-human technosignature?
Our authors were aching to come out with that sentence for the entire, rather long, paper (including, of course, Supplementary pages). But, to their credit, they held off. Though it was there in spirit the whole way through. You could feel it coming.
Before we get to the details, let’s orient ourselves, a step you ought to do when considering any model. What is our goal, our constant ever-present goal? Yes: to find all the evidence probative of the causes of the object under investigation. We want cause. Always cause.
Here: cause of what? Why, of those “transients”, the spots on the photographic plates. What are the possibilities? That list is always an assumption (as we discussed many times). You might not assume the right list, in the sense that you missed the true cause. Or you might have it, but have polluted the list with an enormous glut of distractions, all of which make you lose sight of the cause.
Importantly, there is never a procedure that guarantees you will identify the cause in your list. No, not even with “AI”.
The possibilities as offered by the authors: plate aberrations, balloons, planes, secret satellites (at high orbits), cosmic rays, and, of course, UFOs.
They dismiss satellites for the good reason that their launches would have been noticed, and that this was the pre-satellite era. They dismiss balloons and planes because (and I agree) these would have caused streaks on the plates, which were exposed for hours. The telescopes were focused at infinity, you see. But this also rules out UFOs in any kind of close orbit. Because those UFOs would have had to remain fixed in the sky with reference to the telescope, as the earth rotated for those hours. Stars can do this, because their movement is too small when seen so far away.
On the other hand, you can argue UFOs can do this maneuvering, too, by (say) nice control of their engines, but why would they have picked this particular telescope to remain “fixed to”, when their interest—their motive, a part of the cause—was to monitor the nuke tests? Wouldn’t they rather have “hovered” over those test sites? Or at whatever point was optimal for their investigations? Did they remain fixed in orbit by coincidence with just those hours the plate was exposed?
This seems to rule out close-orbit UFOs. But that leaves far-orbit, “home” or far-base UFOs, stationed out at “infinity”, defined with respect to the telescope. Let that be so. Then there must be an alliance of the sort that Klaatu warned us about in (the original) The Day The Earth Stood Still, monitoring up-and-coming worlds and policing those worlds’ aggressiveness, because why? Because those transients were all over the place. Not the same place. Isn’t that odd?
Why did they come the day before tests? Or rather, why are the authors counting these instances? Did the UFO authorities only have incomplete intel and just guessed wrong? A lot? And did they not know they would be picked up by telescopes (if in far-orbit), but not care? Now the authors note that because of certain technicalities, some of the “day before” might well have been “day of,” so my objection carries less weight. But what happens if we expand the window a day? What if we only look, say, five days before or whatever? My guess is that the same wee Ps would be found in the tests mentioned below.
And then, what about all those other transients, also categorized as “real” by the Machine Learning, that were not in the nuke-test window? What caused them? Were these aliens window shopping on off days, perhaps seeking good candidates for probing? You can claim it is something like this, non-nuke related, but then you have to explain why these signals disappeared once observation techniques improved. We expect they would still be happening if transients were not just nuke related.
Now I trouble you will all these conjectures because the authors did not. They merely tease, and hoping by that tease, to convince, or at least intrigue. This was a mistake on their part, since the goal of their paper was to discover the cause of those transients. (As cause is the goal of any paper.)
If we dismiss bad pictures (aberrations), balloons, planes, satellites (ours), near-orbit UFOs, and cosmic rays (more about that below), we are left with far-off UFOs with really strange behavior, and which have disappeared for some reason in our current era.
I think bad pictures (I include cosmic rays in this) is vastly more likely, which is not proof that the probability of UFOs, on this evidence, is zero. In fact, on this evidence, it is not zero. Just close to it. And remember: the probability is not the decision. It is just a probability. It doesn’t tell you what to believe. Probabilities never tell you what to believe (except about probabilities). You can have a low chance, on this evidence, of UFOs and still believe in them, depending on what that belief means to you.
More on this probability below, because that what this—and really all—papers are about.
Incidentally, you will see things like that discussed in the upcoming second edition of Everything You Believe Is Wrong. Sunday, 22 November.
Details
We can trust (though I did not check) that the authors have made no errors in any of their calculations or data handling. We’ll anyway see, in the end, the P-value has done them the dirty here.
Their transients “persist on sub-second timescales (i.e., brief flashes)”, and diminish by some 30% in earth’s shadow, “consistent with some transients representing highly reflective objects in orbit.” There are many “false positives in existing transient data given their automated identification on scanned photographic plates more than 50 years old likely prone to plate defects (e.g., emulsion errors, dust, scratches)”.
They used a ML model to classify whether “each transient of the n = 107,875 previously identified reflected a real object versus plate defect.” There will be some error in this model, as there are in all models. They ran some hypothesis tests to “confirm” their model, which, as you know, is weak sauce. And anyway a “real” transient only here means one that is likely not a known plate defect. And that’s it.
They say (many) things like “If the ML model were successfully removing plate defects and increasing the relative signal of real transients, the shadow deficit previously reported was expected to be largest in the subset of transients with the highest ML-derived probability scores (Decile 9).” Then they do more hypothesis tests on this. But these are neither here nor there, because “real” transients only mean likely not a plate defect. These models say nothing about any transient’s cause—except the cause is not likely to be a plate defect.
Then the meat: “if the ML model were successfully eliminating false positive transients, the strength of association between nuclear testing and number of transients would increase as the ML-derived probability that transients were real increased”. But this statement assumes the nuke testing causes (in the motive sense) the transients.
Then come the hypothesis tests for this, done in a most confusing way. But it doesn’t matter to us. They have their correlation: the “probability-weighted transient count” of likely “real” transients on days, plus or minus one, there was nuke testing.
The question remains: what caused these transients?
They admit “Atmospheric effects of solar radiation (e.g., Aurora Borealis) or cosmic rays might produce some transients via direct impact on the photographic emulsion or effects of Cherenkov-type phenomena in the upper atmosphere.” The sun might have moderated galactic causes or caused some. But out comes another hypothesis tests with sunspot number and transient count. Which, of course, is not proof the sun caused or didn’t cause any particular transient.
The agree there is a possibility there is “an unknown population of previously undiscovered natural objects in orbit with high specular reflectivity and flat surfaces.”
But how about the nuke testing itself? Some small number of bouncy-ness or scatter from the atmosphere during the tests effecting the plates. I don’t mean bombarded. Just the occasional transient.
Back to Bayes
Here we can use Bayes formula to help us think about the causes.
$$\Pr(\mbox{UFO} | \mbox{TE}) = \frac{\Pr(\mbox{T} | \mbox{UFO,E}) \Pr(\mbox{UFO} | \mbox{E})}{\Pr(\mbox{T} | \mbox{UFO,E}) \Pr(\mbox{UFO} | \mbox{E}) + \Pr(\mbox{T} | \mbox{NC,E}) \Pr(\mbox{NC} | \mbox{E})},$$
where we have two possible causes, UFO and NC = natural causes (to include bad plates, cosmis rays, reflections, etc.), and E is whatever background evidence we assume to build our models. The T is the transient data.
Ignore the “priors” Pr(UFO|E) = 1 – Pr(NC|E). Let’s think only about the data models in the denominator: Pr(T|UFO, E) and Pr(T|NC, E), i.e. the chance we see those transients assuming UFOs are certainly true, and then assuming the Natural Causes are true.
It seems to me (not the authors) these would both be high, or something close to equal. If the Natural Causes are certainly true, then they would have caused the transients. If the UFOs are certainly true, they would have caused the transients, or some of them. If that’s true, then the transients provide almost no discrimination between our two “hypotheses.” And the only thing left to tell difference is whatever else we bring to the problem (those E).
The model Pr(T|UFO, E) is obviously high for the authors. So what they need, to make Pr(UFO|T, E) high, is for Pr(T|NC, E) to be low. To get that, they have to made a big deal of the timing of transients, and say the chance transients occurred in these windows, assuming natural causes to certainly be true, is low. And here they might have been tempted, as many are, by those wee Ps, substituting them in for Pr(T|NC, E). Which is, of course, a mistake.
Even if they did not make that mistake, and genuinely find Pr(T|NC, E) to be low, we do not have to join them in their judgement. And, for all the reasons I gave above, I do not.
And that is how you ought to examine all papers.
All my work is free and user-supported. Here’s how to help:
- Subscribe at Substack (paid or free!)
- Cash App: $WilliamMBriggs
- Zelle: use email: matt@wmbriggs.com
- Buy me a coffee
- Paypal
- Other credit card subscription or single donations
- Hire me
- Subscribe at YouTube
- PASS POSTS ON TO OTHERS
Discover more from William M. Briggs
Subscribe to get the latest posts sent to your email.
