If we zoom out a little, what is an LLM trying to accomplish mathematically? Or better yet,

Ques. What sort of morphing happens to the input information given to the blackbox we call LLM?

This blackbox interacts with information in two different phases: during training and inference. Just know that the morphing it does to the information during training time is a superset of that during inference.

Ans. LLM is trying to minimize conditional entropy, the real goal of all intelligent systems.

---

Observing a system means observing some property of that system. One very usefull such property is Entropy. Let's quickly go over Entropy, Conditional Entropy, Encoding, and Optimal Encoding.

Conditional Entropy:

In information theory, "Entropy" tells you how unpredictable something is, on average.

Entropy $H(Y)$ is the expected number of bits required to encode the outcome of a random variable under an optimal encoding scheme.

• $H(Y) = -\sum_{y \in \mathcal{Y}} p(y) \log p(y)$

Conditional Entropy:

Conditional entropy $H(Y|X)$ measures the uncertainty remaining in $Y$ after observing $X$: $H(Y|X) = -\sum_{x \in \mathcal{X}} p(x) \sum_{y \in \mathcal{Y}} p(y|x) \log p(y|x)$

It can also be expressed as: $H(Y|X) = H(X,Y) - H(X)$

where $H(X,Y)$ is the joint entropy.

Minimizing Conditional Entropy(LLM):

An LLM can be viewed as a communication channel where:

• Input $X$: tokens from the training corpus
• Output $Y$: next token predictions
• The model parameters $\theta$ define the channel $p_\theta(y|x)$

The training objective minimizes the cross-entropy loss, which is equivalent to minimizing the conditional entropy $H(Y|X_\theta)$ where $X_\theta$ represents the model's internal representation of the input.

However, this view is incomplete. The true information-theoretic objective involves finding an optimal balance between compression and prediction - this is where the Information Bottleneck principle becomes essential.

Information Bottleneck Principle

The Information Bottleneck (IB) principle, introduced by Tishby et al. (1999), frames learning as a trade-off between two competing objectives:

1. Compression: Minimize the information the retained representation $T$ carries about the input $X$
2. Prediction: Maximize the information the retained representation $T$ carries about the relevant output $Y$

Formally, we seek to optimize the IB Lagrangian: $\mathcal{L}_{IB}[p(t|x)] = I(X;T) - \beta I(T;Y)$

where:

• $I(X;T)$ is the mutual information between input and representation (to be minimized)
• $I(T;Y)$ is the mutual information between representation and target (to be maximized)
• $\beta \geq 0$ is a Lagrange multiplier controlling the trade-off

For $\beta \rightarrow \infty$, we preserve all information about $Y$ (pure prediction) For $\beta \rightarrow 0$, we compress $X$ as much as possible (pure compression)

In the context of neural networks, each layer learns a stochastic mapping $p(t^{(l)}|x)$ from the input to its representation. Training via SGD on the cross-entropy loss implicitly optimizes an IB objective where $\beta$ is related to the inverse temperature in statistical physics formulations.

Channel Capacity Interpretation

Now we connect this to channel capacity. Consider the neural network as a communication channel:

• Input: Training samples $X \sim p_{data}(x)$
• Channel: The stochastic mapping defined by network weights $p_\theta(t|x)$
• Output: Network representation $T$
• Noise: Induced by the stochastic nature of learning (SGD noise, dropout, etc.)

The channel capacity $C$ of this stochastic mapping is defined as: $C = \max_{p(x)} I(X;T)$

This represents the maximum rate at which information can be reliably transmitted from input to representation through the network.

Channel capacity view of LLM training through the information bottleneck.

During training, we're not merely minimizing $H(Y|X)$ but rather operating near the channel capacity limit while preserving predictive information. The IB principle tells us that optimal learning occurs when we operate at: $I(X;T) = C \quad \text{and} \quad I(T;Y) = \text{maximum possible given } C$

This creates an interesting dynamic: as training progresses, the network initially increases $I(X;T)$ (fitting phase) then decreases it (compression phase), all while trying to maximize $I(T;Y)$.

Saturation and the Limits of Learning

The concept of saturation emerges naturally from this framework. When we say a network has "saturated" in its learning capacity, we mean it has approached the channel capacity limit: $I(X;T) \approx C$

Beyond this point, additional training cannot increase the mutual information between input and representation. Any further reduction in loss must come from better utilization of the fixed information budget - i.e., increasing $I(T;Y)$ for a fixed $I(X;T)$.

This explains several empirical phenomena:

1. Plateaus in training loss: When $I(X;T)$ hits capacity, further optimization focuses on representation quality
2. Double descent: As model capacity increases, so does channel capacity $C$, allowing better fits
3. Generalization bound connection: The IB framework provides tighter generalization bounds than VC-dimension or Rademacher complexity approaches

In the next section, we'll connect these abstract information quantities to measurable properties of neural networks and derive practical implications for architecture design and training protocols.

Empirical Connections and Practical Implications

The channel capacity framework provides powerful explanations for several empirically observed phenomena in deep learning:

1. Scaling Laws and Channel Capacity

Recent work has shown that language model performance follows predictable power-law scaling with respect to model size, dataset size, and compute. In our framework:

• Model capacity directly relates to the maximum achievable channel capacity $C_{max}$
• Dataset size affects the empirical distribution $p_{data}(x)$ we're trying to model
• Compute determines how closely we can approach the theoretical channel capacity

The scaling laws can be interpreted as describing how close finite-trained networks come to the information-theoretic limit: $\text{Performance} \approx f\left(\frac{I(X;T)}{C}\right)$

where $f$ is some monotonic function and $I(X;T)/C$ represents the utilization ratio of the available channel capacity.

2. Emergent Abilities as Phase Transitions

The sudden emergence of abilities at certain scale thresholds can be viewed as phase transitions in the information plane. As model size increases:

1. Channel capacity $C$ increases continuously
2. At critical thresholds, new discrete representations become encodable within the channel
3. This leads to discontinuous jumps in $I(T;Y)$ for specific tasks

This explains why abilities like multi-step reasoning or tool use appear abruptly rather than gradually.

3. The Compression Phase and Generalization

The observed compression phase during training (where $I(X;T)$ decreases after peaking) aligns with the IB theory's prediction that optimal representations discard input information irrelevant to the task. Networks that fail to compress properly often exhibit:

• Memorization rather than generalization
• Poor out-of-distribution performance
• Sensitivity to input perturbations

Practical Implications for Architecture Design

Understanding neural networks through the channel capacity lens suggests several design principles:

1. Capacity Matching: Architecture should be chosen such that its potential channel capacity matches the complexity of the target distribution
2. Information Flow Optimization: Skip connections and normalization layers can be viewed as mechanisms to preserve information flow across layers
3. Regularization as Capacity Control: Techniques like dropout, weight decay, and early stopping effectively modify the channel's noise characteristics, thereby controlling usable capacity
4. Training Protocol Design: Learning rate schedules can be interpreted as annealing protocols that allow the system to explore the information plane optimally

Conclusion

Reframing neural network training through the lens of channel capacity and the Information Bottleneck principle provides a unified framework for understanding:

• Why deep learning works despite being vastly overparameterized
• How scaling laws emerge from fundamental information limits
• What constitutes optimal learning in finite-sample regimes
• How to diagnose and improve training dynamics

The most powerful insight is that intelligent systems don't merely minimize prediction error—they navigate the complex trade-off between representation sufficiency and minimality, constantly operating near the limits of what their architectural channels can reliably transmit.

This perspective shifts our focus from empirical heuristics to information-theoretic first principles, offering a more robust foundation for advancing both our theoretical understanding and practical capabilities in AI.

---

Takeaways:

Your ability to think about intelligence will(already have) develop(ed) unhealthy level of dependence on the attention like mechanisms, and LLM like systems. The only way to break free and progress humanity faster is to leave the dear knowledge you've grown comfort around.

Only keep the information theory level understanding close to your heart and that won't harm your ability in any way possible. It let's me stay sharp, creative, and agile.