Reminder: The Thursday Class is only for those interested in studying uncertainty. I don’t expect all want to read these posts. Pease don’t feel like you must. Yet, I have nowhere else to put them. Your support makes this Class possible for those who need it. Thank you. MUCH MUCH MATH ALERT!
Ignorance has consequences. Good ones, in this case. Now I have tried my best to reduce the amount of math needed to follow these lectures, emphasizing the philosophy. Today a certain amount of math becomes unavoidable. Still, if you have trouble with it, I urge you to “read around” it for the philosophy.
Video
Links: YouTube * Twitter – X * Rumble * Bitchute * Class Page * Jaynes Book * Uncertainty
HOMEWORK: Read Jaynes Chapter 11.
Lecture
Review, I beg you, Entropy Part I. I won’t repeat any of that lesson here.
Except this: recall our goal: to deduce models given whatever assumptions we entertain, knowing that the best model is that which identifies all causes and conditions which make our proposition of interest true (or false). And since we rarely have that, the next best is The Correlation. Let’s see.
Deduce the model given by this evidence: You know nothing about the proposition of interest (POI); you are entirely ignorant.
If you are a classical frequentist modeler, you might say “Let’s use a normal distribution, with these two parameters, which we can estimate.” If you are a Bayesian you will add, “I want to tighten this by adding non-informative priors on those parameters.”
Frequentist: “You can’t do that. You can just add information about these real (they exist in Nature, you know) parameters. Your subjectivity will given undue weight to your priors, unlike my objective model, which is backed by any number of central limit theorems.”
Bayesian: “I understand your concern, but the effect of my priors washes away in the long run.”
If you are a Probabilist (as we in the Class all are), we say, “You guys are like those physicists who try to explain how something can arise from nothing by calling ‘nothing’ quantum fields with some large list of properties. Quantum fields are not nothing: they are something. The evidence is we know nothing; therefore, the only deduction is no probability at all.”
And there we have cleared away the “battle” between frequentists (who are wrong, as we saw, for a host of other reasons) and Bayesians.
Suppose next all we know—where that all is as strict as I can emphasize, and which even though I shout it from the top of the Washington monument will still be taken by some as “not all”—is that our POI can take, and must take, of two values. Yes, our old friend the dichotomy. Then we have earlier deduced the model Laplace’s Rule of Succession (I gave the formula last time). Laplace starts, before any observations, with uniformity; i.e. equal probability for either possibility. Don’t forget we proved the validity of Laplace through Stove’s statistical syllogism (one of the first lessons).
Now suppose all we know is that our POI can take k possible values (k > 0). We deduce that we must begin with probabilities that maximize entropy. I.e.
$$H|E = -\sum_i^k p_i \log(p_i).$$
And the unique p_i that max this are p_i = p for all k.
What happens after observations come in? Well, that is an excellent question. Something like a multinomial, or its posterior predictive distribution, like a Dirichlet distribution. At the limit, that is, after we have seen many observations.
$$\Pr(y|xE) = \binom{m}{y}\frac{B(x+1+y)}{B(x+1)},$$
where the y, x, and “1” are all of the dimension k, the same as the POI (two for a dichotomy, three for trichotomy, etc,), and where x is the old , and the B are old-fashioned multivariate beta functions. For instance,
$$B(\alpha) = \frac{\prod_{i=1}^k \Gamma(\alpha_i) }{\Gamma(\sum_{i=1}^k \alpha_i)},$$
and the upsidedown L is the gamma function.
We observe counts x = (x_1, x_2, …, x_k), i.e. the number in each “bucket” x_i. And we want to know the probability of y = (y_1, y_2, …, y_k), where we have to specify how many in each bucket we might see, which is our POI. Note that “x + 1” here is (x_1, x_2, …, x_k) + (1, 1, …, 1).
Let
$$\sum_i x_i = n,$$
then if we are taking just one more observation after seeing x,
$$\Pr(y_i = 1 | xE) = \frac{x_i + 1}{n+k}.$$
This works for the limit of Laplace’s rule of succession, too, because there k = 2. But what happens if we’re before the limit, or not anywhere “close” enough to it, the approximation given above doesn’t look beautiful?
I don’t know. This is an open problem, to redo Laplace’s Rule of Succession for k > 2. (I’m surprised Jaynes never did it. I’m not so surprised I didn’t, since he’s a better mathematician than I can ever be. But I’m hoping you can.)
We also violated our problem, just as badly as the Bayesian or frequentist, by adding information we were not supplied. The information was, you recall, only that the POI could take (here) k values. That’s it. Nothing more. That’s the finite version that leads to Laplace when k = 2. We cheated, though, and said, like everybody says, “Why don’t we invoked this ad hoc parameterized probability model that sounds like it might work? Then we’ll use the maximum entropy rule on those continuous (i.e., at the limit) parameters. See what happens.”
We saw what happens. It’s a plausible answer. But there is no escaping our cheated. We cheated. We added information. There was no logical way to get to the ad hoc parameterized model from the information. None. We are thought criminals.
This isn’t hyperbole or some literary trick where I’m going to reveal that, actually, what we did was fine. Because it wasn’t. We are bad people.
In this case, I think we guessed right, but it’s only a guess. I don’t have a proof. That would be the proof of Laplace for k > 2. Which, as I said, I do not have.
Now we can use the excuse we had to cheat, because how else can we get an answer unless we cheat? That is an ethics question, and not a mathematical one. Yet it is the same question asked and answered daily millions (I likely do not exaggerate) of times a day when some researcher uses some ad hoc parameterized model that goes beyond the evidence supplied.
And he forgets, or they forget, those researchers. They have cleaned their memories of their deed and find themselves blameless. Yet there is no escaping it.
All probability is, as I have reminded us so many times that we’ll have to invent a new number to describe, conditional on the information we assume. And you know what they say about assume. But what it also means is that once I have added information to the problem, even though I have cheated in doing so, I can calculate whatever probabilities I like. Probability (logic) is not responsible for my assumptions. Probability, and therefore models, are mute on this important subject.
Remember this, too: in Laplace, when we go to the limit, the parameters emerge. That is the origin of parameters, you recall, approximations to our finite discrete measurements of Reality at the limit. Which we can never reach.
What this means is that nearly every (or every?) model in the scientific literature that uses parameters is adding information that isn’t there in Reality, or that is only a guess of what Reality might be, and is anyway always ever an approximation to our fine discrete measurementss. Which is yet another reason for the over-abundance of over-certainty in science.
Yet once we have made our thought crime, like any common frequentist or Bayesian, and posited an ad hoc parameterized model, we are free to use the maximum entropy principle. That’s what I did.
I used the information that gave us the POI and it’s k levels, and figured that would be fit into an ad hoc parameterized multinomial model. I guessed that since, at the limit, that is what Laplace turned out to be for k = 2, so it might also work for k > 2. Does it? I don’t know.
But that multinomial does come with parameters, usually called alpha, for each of the “buckets”, and these have as possible values any number from 0 to 1 (at the limit!, this the real line). These parameters are usually, and egregiously, and weepingly, called the “probabilities” of those buckets. The humanity! They are not. They are nothing but poor pitful parameters. That they are called “probabilities” is yet another reason there is so much confusion about what probability is. Never make that mistake.
You will, though. Because everybody gives life to parameters. It ‘s a sad thing.
Anyway, it’s clear enough that “maxent”, in entropy’s continuous form, gives uniform values for each alpha, i.e. an equal chance to be any number between 0 and 1. This is called, also a lousy name, an “ignorance” “prior”, or words to like effect. There is no ignorance about it. We assumed the parameters! And once we did, and assumed we knew nothing else about them, maxent gave us the answer.
With that answer, we can continue as with any probability, and out pops the “predictive posterior”. Which is to say, the deduced probability model for y given our assumptions.
And that is all the maxent principle is.
It ought to go without saying, but a lot of things that ought to be aren’t, so I’ll say it: if you have other information about the problem other than just that the POI can be k possibilities, then that information should be part of your probability model. Which it often isn’t. The thought crimes are on both sides of probability.
But that’s for another day.
All my work is free and user-supported. Here’s how to help:
- Subscribe at Substack (paid or free!)
- Cash App: $WilliamMBriggs
- Zelle: use email: matt@wmbriggs.com
- Buy me a coffee
- Paypal (try $1 a month!)
- Other credit card subscription or single donations
- Hire me
- Subscribe at YouTube
- PASS POSTS ON TO OTHERS
Discover more from William M. Briggs
Subscribe to get the latest posts sent to your email.
