What is Entropy?

What Is Entropy?

The thread centers around the elusive concept of entropy, sparked by John Carlos Baez’s article explaining entropy across disciplines—information theory, statistical mechanics, and epistemology. The discussion opens with Claude Shannon’s famous anecdote involving John von Neumann, who suggested naming Shannon’s uncertainty function “entropy” because “no one really knows what entropy really is.”

Several participants argue that entropy is fundamentally subjective, emphasizing that it depends on the observer’s knowledge about a system. For example, the entropy of a variable X should be annotated H_observer(X), implying it reflects the observer’s uncertainty. This view is extended by users who cite cross-entropy and KL divergence as tools to relate belief distributions (subjective) to actual distributions (objective). Supporters of this perspective highlight how entropy can be decomposed into aleatoric uncertainty (inherent randomness) and epistemic uncertainty (stemming from incorrect beliefs), thus framing entropy as both a model-dependent and observer-dependent quantity.

Conversely, critics assert that entropy can be rigorously defined as an objective property of a distribution, independent of belief. One user notes that entropy in physics—particularly Shannon and Gibbs entropy—is calculated over known distributions, such as equilibrium states, which are independent of individual beliefs. Another contributor underlines this by referencing how physical systems (e.g. quantum superpositions or GPT-2’s output probabilities) possess mathematically defined distributions whose entropy does not stem from personal uncertainty.

The debate then evolves toward whether different observers—given varying access to information—might assign different entropies to the same physical system. Examples include random number generators and dice with hidden internal states, suggesting that the completeness or granularity of one’s model affects entropy calculation. Some users emphasize that coarse-grained or incomplete sample spaces limit a subject’s ability to infer underlying dynamics, reinforcing the role of modeling choices in shaping entropy.

Further exploration touches on broader epistemological themes, such as the limits of human understanding and the nature of induction. Solomonoff induction is introduced as a theoretical framework that blends prior complexity with observational consistency, illustrating an idealized approach to inference. Discussions on abstraction levels, sample space evolution, and Bayesian versus frequentist views enrich this dialogue, pointing to entropy’s intersection with learning, modeling, and belief revision.

Throughout the thread, von Neumann is revered for his multidisciplinary genius, with colorful anecdotes about his intellect and influence across math, physics, and computing. His suggestion to use "entropy" as a strategic rhetorical tool is seen as prescient, as it continues to challenge thinkers across fields.

#EntropyDebate #ShannonEntropy #SubjectivityInScience #VonNeumannLegacy