Information Entropy

CLN Wiki · Theory for #information

Information Entropy CORE

= measure of uncertainty @ Shannon (1948)

SUMMARY click to copy as CLN

=Information Entropy — the average amount of surprise or uncertainty in a message

.Higher probability = Lower entropy (less surprise)

.01 · Interpretation

→Entropy = Information Content

.02 · Application

→CLN maximizes entropy per token

Surprise is information. Double-click any CLN line to anchor it. Right-click to tagmarry or record a decision.

01// Definition Term

// ##IE-DEF

=Information Entropy — a quantitative measure of the uncertainty or randomness in a message

.Introduced by Claude Shannon in "A Mathematical Theory of Communication"

→Formula: H = -Σ p(x) log p(x)

!A rare event carries more information (higher entropy) than a common one

02// Interpretation Logic

// ##IE-INT

*Low Entropy

.High predictability, low surprise

-Example: "The sun will rise tomorrow" (Redundant)

*High Entropy

.Low predictability, maximum surprise

+Example: Random noise or a lottery result

☐"Umm, well, basically..." (Entropy ≈ 0)

│Information Scale

▣"Entropy = Surprise" (Entropy = Max)

03// Application Analysis

// ##IE-APP

.Natural Language has low entropy (predictable grammar, filler words)

.CLN increases entropy per token by removing predictability

→The Paradox

~We want high information (entropy) but low noise (randomness)

.CLN solves this by creating structured surprise (operators acting as semantic concentrators)

04// Implication Implication

// ##IE-IMP

~Entropy defines the limit of lossless compression

.Shannon's Source Coding Theorem

+To learn is to reduce the entropy of the unknown

// ##IE-OPEN

?Does consciousness require high entropy processing?