Information Theory and Coding (UCB FA24 EE229A, rewritten in 2025)

INFOTH Note 5: Fano Ineq., AEP

Fano's Inequality, Markov Chain, Entropy Rate, Asymptotic Equipartition Property and Lossless Compression

INFOTH Note 4: Cond. MI & Cond. KL, Data Processing Ineq.

Conditional Mutual Information, Conditional Relative Entropy, Chain Rule for these things, Properties, and Data Processing Inequality

INFOTH Note 3: Relative Ent. (KL), Mutual Info.

Log-sum Ineq., Relative Intropy (aka. KL Divergence) and Mutual Information

INFOTH Note 2: Joint & Conditional Entropy, Chain Rule

Definition of Many Entropies and the Chain Rule. We have a specific def for joint entropy.

INFOTH Note 1: Axioms & Def of Entropy

Shannon Entropy, Axioms, Entropy and Concavity