Class 99: Entropy Part I

Reminder: The Thursday Class is only for those interested in studying uncertainty. I don’t expect all want to read these posts. Pease don’t feel like you must. Yet, I have nowhere else to put them. Your support makes this Class possible for those who need it. Thank you. MUCH MUCH MATH ALERT!

Entropy is a lovely and useful concept, but one apt to be misunderstood. In the end, probability rules. The topic is large. Today, only a brief introduction, which is not, in the end, mostly about entropy. Instead it is a caution against misuse.

Video

Links: YouTube * Twitter – X * Rumble * Bitchute * Class Page * Jaynes Book * Uncertainty

HOMEWORK: Read the paper!

Lecture

Earlier we together read Bryan Johnson Immortality manifesto, in which he makes a big to-do about entropy. Entropy is, in both forms, physics and information, a tremendously useful tool. Please do not go away from the criticisms below (like last week about over-certainty in genetic analysis) with that idea “Briggs doesn’t believe in entropy!” (or “genes!”). Briggs instead likes to put things in their proper perspective.

There is nothing special about Johnson’s use of entropy; many make the same errors he does. But since his manifesto might still be fresh in your mind, we can use his example:

Additionally, living systems are not defined only by matter, but by organization, so we look to a second formulation of entropy that comes from information theory:

$$H = – \sum_i p_i \log p_i$$

where H is Shannon entropy and $p_i$ are the probabilities of possible system states.

DNA sequence, epigenetic state, protein concentrations, tissue architecture, neural connectivity, and dynamic regulatory loops all encode biological information. A living organism is therefore not merely a pile of molecules, but a highly structured informational process…

I’ve reminded us, many times, about the dangers of notation, and here we meet one of them. It is not “H”, but more properly “H|E”, where the evidence E, upon which we base the probabilities, is denoted. You almost never see this more rigorous notation, except in “conditional” entropy, which is akin to “conditional” probability in classic texts. But all probability, and therefore all entropy, is conditional. Nothing has a probability, including DNA. There is only probability with reference to evidence (assumed).

Entropy is a measure of information, like probability is; indeed, entropy is only a function of probability, and all probability is conditional on all the evidence we assume. Which means entropy is in our minds, and not in things. That DNA “encodes information” is a metaphor, one that is most useful, but it remains a metaphor.

This means we have everything we need in probability itself. But, of course, sometimes functions of probability have various nice uses. And so it is with entropy. Yet never fall into the gross error that it offers something more than probability.

As I said in the Johnson critique, there are two notions of entropy, the original one from statistical physics and the informational one put forward by Claude Shannon (who, I am proud to say, is from the same town I am). The mathematical form is the same. Shannon was after a one-number summary of a probability distribution. We all remember the dangers of one-number summaries: they can be helpful as long as we recall they necessarily strip away information. Here that has an ironic meaning.

Shannon’s idea (some say) was to quantify the numerical measure of “surprise” in a probability; or, rather, in the proposition the probability represented. His idea was to take -log_2(p_i) as that measure: an almost monotonic function of probability. Smaller p_i, larger surprise. What happens when p_i = 0? Well, infinite surprise; and a discontinuity. But that’s not quite right, is it, because probabilities don’t happen: things do. A model can say Pr(Y_i | E) = p_i = 0, and yet we observe Y_i. The p_i is still not “infinite” surprise because we said at the beginning p_i = 0, though the model has been falsified.

This makes the “surprise”, well, a surprising word. Suppose a model gives positive probability to five colors (for some object): Red, Green, Blue, Yellow, White. From this we imply that it has zero probability, and thus an infinity of infinite surprises, for every other color. Every model is like this, because no model gives positive probability for everything in the world that can happen. So every model evinces infinite surprise.

Which means “surprise” is the wrong word here. It’s being applied to things, objects, observations, and not probabilities. There can be no surprise in a probability. Probabilities, after all, are deductions (based on the evidence you assume). We know probabilities. But events can surprise us. So there’s a strange conflation of probability and outcome, or frequency. Yet probabilities are not frequencies. Let’s see where this goes.

The log base 2 is not a necessity, incidentally, but it’s natural when things are encoded in binary language, and you know where that happens. And indeed information is a far superior word to surprise, and was Shannon’s original goal. Which in our notation was this:

$$Pr(Y = y | s E),$$

where y is some character or encoding or the like, and s is some received signal, and E all the other information that goes into our model, including the idea that signal may be corrupted by true noise. Refresh your memory over the difference between true noise and faux noise, as it is treated in many time series models.

At any rate, this is a fine model form or goal because this is, after all, how you are seeing these words. Your robot (or rather Jaynes’s robot) is receiving signals over a channel and must decide—a word which tells us we also have decision functions and not just probability—which characters to display. A simple decision rule is “display the character y with the highest Pr(Y=y|sE).” (AI is not unlike this!)

The average, or rather “expected value”, of surprise (or what we’re calling surprise), or rather of information, is this:

$$H = – \sum_i p_i \log p_i$$.

If the log is any other base than 2, sometimes a constant (k, usually, because of statistical physics) is prefixed, but of course isn’t needed for information. So entropy is yet another “expected” value. But it has some “nice properties”, as we mathematicians say. Keeping in mind always that everything is already in the probabilities which we found by conditioning on whatever evidence we assumed. Entropy does not exist separately from that.

One nice property is that the entropy is maximized when, for all i, p_i = p. That is, when each proposition in your range of possibilities has, conditional on your evidence assumed, equal chance. I apologize for continuously repeating the “conditional on your evidence assumed” bit (bit? bit? entropy? get it? get it?), but if I do not, then like the P-value, entropy has a danger of coming alive in your mind. Always a terrible mistake. More: If there are n possibilities, p_1, p_2, …, p_n, then as n increases, H increases (or does not decrease). And there are other niceties we will explore later.

As with the characters (these words) you are seeing right now, we need to discover a way to use past observations to form our model, which gives us our p_i, which we get, one for every character our channel was built to handle. The decision function comes from our desires and goals, like all our decision functions. For now, we’ll suppose we’re using the simple one (show the character with max p_i).

How do we move from these observations to models? One way, and the simplest, is to suppose the frequency we observed some character is then modeled as its probability. A fine idea, many times. But not all times. And one which we have seen, all through history, is abused, when the probability becomes the frequency, which is backwards. With many bad consequences, which we covered before.

We (at long last) return to Jaynes, referring to the difference between probability and frequency. Here is a quote from his Chapter 9.10, to which we fully assent:

In our terminology, a probability is something that we assign, in order to represent a state of knowledge, or that we calculate from previously assigned probabilities according to the rules of probability theory. A frequency is a factual property of the real world that we measure or estimate. The phrase ‘estimating a probability’ is just as much a logical incongruity as ‘assigning a frequency’ or ‘drawing a square circle’.

The fundamental, inescapable distinction between probability and frequency lies in this relativity principle: probabilities change when we change our state of knowledge; frequencies do not. It follows that the probability p(E) that we assign to an event E can be equal to its frequency f (E) only for certain particular states of knowledge. Intuitively, one would expect this to be the case when the only information we have about E consists of its observed frequency; and the mathematical rules of probability theory confirm this in the following way.

The “following way” leads us to entropy, among other ideas. We want some method to tie observations to probabilities. In other words, we want a model. Probabilities are models. Can we deduce—not just guess, but deduce—probability models from observations? We already did so, as Jaynes reminds us, using Laplace’s rule of succession.

Incidentally, here’s the R code for that again (after observing k.old “1”s in n.old observations, of some dichotomy, and where you want the probability of x new “1”s in n.new to-come observations):

newdbinom = function(x, n.new, k.old, n.old){
   # New observables for binomial distribution given old data and guess of how many new data points there will be n.new
   # beta-binomial
   a = k.old+1
   b = n.old-k.old+1
   (ans = exp(lchoose(n.new,x)+lbeta(x+a,n.new-x+b)-lbeta(a,b)))
}

Now we come to the derivation of H; or, rather, Jaynes does, in Chapter 11, which I urge you to read. It is not that difficult. We won’t go through that here, since are after understanding. Recall Jaynes’s original goal was to have a machine or robot programmed to take all available information and assign probabilities based on that information, in a completely logical manner. Laplace’s Rule is one of these ways, but is useful only when there are two possible outcomes (that dichotomy), and we know nothing more than one or the other is a possibility. And where that nothing is strict, strict, strict.

If we have multiple outcomes, not just two, and all we know is that each outcome is a possibility, then it turns out that the p_i that maximize the entropy H gives us the deduced probabilities with this information. The base on the log is, however, arbitrary, but any choice would only change H by a multiplicative constant (that k). And so isn’t needed.

Page 351:

The function H is called the entropy, or, better, the information entropy of the distribution {p_i}. This is an unfortunate terminology, which now seems impossible to correct. We must warn at the outset that the major occupational disease of this field is a persistent failure to distinguish between the information entropy, which is a property of any probability distribution, and the experimental entropy of thermodynamics, which is instead a property of a thermodynamic state as defined, for example by such observed quantities as pressure, volume, temperature, magnetization, of some physical system. They should never have been called by the same name; the experimental entropy makes no reference to any probability distribution, and the information entropy makes no reference to thermodynamics. Many textbooks and research papers are flawed fatally by the author’s failure to distinguish between these entirely different things, and in consequence proving nonsense theorems.

One might reply to that last line, Many such cases.

On the other hand, after much more work, Jaynes states (p. 364) the entropy “relations are all just the standard equations of statistical mechanics given to us by J. Willard Gibbs, but now in a disembodied form with all the physics removed.”

We still need to how to use maximum entropy, or “maxent”, to assign—which is to say, deduce—probabilities in other situations. But this is enough for now.

All my work is free and user-supported. Here’s how to help:

• Subscribe at Substack (paid or free!)
• Cash App: $WilliamMBriggs
• Zelle: use email: matt@wmbriggs.com
• Buy me a coffee
• Paypal
• Other credit card subscription or single donations
• Hire me
• Subscribe at YouTube
• PASS POSTS ON TO OTHERS