Information Is Not Entropy,
Information Is Not Uncertainty!

Thomas D. Schneider

color bar

There are many many statements in the literature which say that information is the same as entropy. The reason for this was told by Tribus. The story goes that Shannon didn't know what to call his measure so he asked von Neumann, who said `You should call it entropy ... [since] ... no one knows what entropy really is, so in a debate you will always have the advantage' (Tribus.McIrvine1971).

Shannon called his measure not only the entropy but also the "uncertainty". I prefer this term because it does not have physical units associated with it. If you correlate information with uncertainty, then you get into deep trouble. Suppose that:

information ~ uncertainty

but since they have almost identical formulae:

uncertainty ~ physical entropy
so

information ~ physical entropy

BUT as a system gets more random, its entropy goes up:

randomness ~ physical entropy

so

information ~ physical randomness

How could that be? Information is the very opposite of randomness!

The confusion comes from neglecting to do a subtraction:

Information is always a measure of the decrease of uncertainty at a receiver (or molecular machine).

If you use this definition, it will clarify all the confusion in the literature.

Note: Shannon understood this distinction and called the uncertainty which is subtracted the 'equivocation'. Shannon (1948) said on page 20:

R = H(x) - Hy(x)

"The conditional entropy Hy(x) will, for convenience, be called the equivocation. It measures the average ambiguity of the received signal."

The mistake is almost always made by people who are not actually trying to use the measure. As a practical example, consider the sequence logos. Further discussion on this topic is in the https://alum.mit.edu/www/toms/bionet.info-theory.faq.html under the topic I'm Confused: How Could Information Equal Entropy?

For a more mathematical approach, see the Information Theory Primer.

Some questions and answers might make these isues more clear.


References Examples of the error
See also Pitfalls in Information Theory and Molecular Information Theory

color bar Small icon for Theory of Molecular Machines: physics,
chemistry, biology, molecular biology, evolutionary theory,
genetic engineering, sequence logos, information theory,
electrical engineering, thermodynamics, statistical
mechanics, hypersphere packing, gumball machines, Maxwell's
Daemon, limits of computers


Schneider Lab

origin: 1997 January 4
updated: 2020 Mar 27: Rename Tribus1971 to Tribus.McIrvine1971

color bar