Information Theory and Coding Notes (UCB FA24 EE229A) on itu

INFOTH Note 23: Parallel Gaussian Channels, WSS, Distributed Source Compression

Tue, 25 Nov 2025 11:45:14 +0800

Reference: T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Ch. 9 (Gaussian Channel), especially Sec. 9.4 (Parallel Gaussian Channels). The proof sketch uses the maximum entropy theorem (Thm 8.6.5) and the entropy chain rule inequality (eq. 8.63).

23.1 Recall: AWGN Capacity

The single-use AWGN channel \(Y = X + N\), \(N \sim \mathcal N(0,\sigma^2)\), \(X \perp\!\!\!\perp N\), has capacity

\[ C = \frac{1}{2}\log\!\left(1 + \frac{P}{\sigma^2}\right) \text{ bits per channel use}, \]

where \(P\) is the per-use power constraint \(E[X^2] \le P\). Communication at rate \(R < C\) is achievable with \(P_e^{(n)} \to 0\); rate \(R > C\) is not (see Note 22).

This is also the mutual information maximizer:

INFOTH Note 22: AWGN Channel and Shannon Limit

Thu, 20 Nov 2025 11:45:14 +0800

22.1 From DMC to Waveform Channels

Until now, we have met DMCs which are discrete, memoryless channels.

While the real world communication be like: a message \(m\) is mapped to (corresponded to) an input waveform \(x(t)\), not discrete samples.

\[ x_m(t)=\sum_{i=1}^n x_i\, g(t - iT_0), \]

where \(x_i\) are symbols and \(T_0\) is the symbol interval.

Waveform Channel Model

\[ x(t)\to\boxed{\text{Channel}}\to y(t)=x(t)+N(t), \]

\(N(t)\) is a continuous-time stochastic process (the additive noise), i.e. \((N(t), t\ge 0)\) or \((N(t), t\in \R)\).

To discuss capacity in a Shannon sense, we consider communication over a fixed time \(T\). Without any constraint, the input \(x(t)\) could carry infinite information (infinite amplitude, infinite bandwidth). We impose two physical constraints:

INFOTH Note 21: Polar Codes and Related Theories (Bhattacharyya, Martingale)

Tue, 18 Nov 2025 11:45:14 +0800

Reference: E. Arikan, “Channel Polarization: A Method for Constructing Capacity-Achieving Codes for Symmetric Binary-Input Memoryless Channels,” IEEE Trans. Inform. Theory, vol. 55, no. 7, pp. 3051–3073, July 2009.

21.1 Core Idea of Polar Codes

Notation. Throughout this note, \(I(W)\) denotes the symmetric capacity \(I(X;Y)\) under uniform input \(X\). For BISO channels, uniform input achieves capacity, so \(I(W)=C(W)\).

本文中 \(I(W)\) 始终指 uniform 输入下的对称容量 \(I(X;Y)\)。对于 BISO 信道，uniform 输入恰好达到容量，因此 \(I(W)=C(W)\)。
Starting point. We have a noisy channel \(W\) with capacity \(I(W)\). Shannon’s channel coding theorem says reliable communication at rate \(I(W)\) is possible, but does not give a practical construction.

有一条噪声信道 \(W\)，容量 \(I(W)\)。Shannon: 能以 \(I(W)\) 的 rate 可靠通信，但没有给出实际的构造方法。
Arikan’s idea. Take \(N=2\) copies of \(W\). Instead of using them independently, mix the inputs through a linear pre-processing step: \(x_1=u_1\oplus u_2\), \(x_2=u_2\), a butterfly transform analogous to the FFT.

使用 \(N=2\) 份 \(W\)，把输入 \((u_1, u_2)\) 经线性变换得到 \((x_1, x_2)\)：\(x_1 = u_1 \oplus u_2,~~ x_2 = u_2\)，这是一个类似 FFT 的蝴蝶变换。
Channel splitting. After mixing, the decoder wants to recover \((u_1,u_2)\) from the outputs \((y_1,y_2)\). Rather than the intuitive order of decoding \(u_2\) first (since \(x_2=u_2\) directly), we deliberately decode \(u_1\) first and then \(u_2\), which enables polarization.

混合后从解码端看，我们想通过输出 \((y_1,y_2)\) 解码 \((u_1,u_2)\). 我们故意设计先不解码直观的 \(u_2\) 而是先解码 \(u_1\)，再解码 \(u_2\)，这样才能极化两个信道。

The two identical physical channels \(W\) can be viewed as two different synthetic sub-channels \(W'\) and \(W''\):

原来的两个相同物理信道 \(W\) 可以看成两个不同的合成子信道 \(W'\) 和 \(W''\)；
- In \(W'\), the sole input is \(u_1\), while \(u_2\) is unknown, an uncontrollable random variable that must be summed over (marginalized out). The presence of this unknown \(u_2\) degrades the decoding of \(u_1\).
  
  在 \(W'\) 我们假设输入只有 \(u_1\)，而 \(u_2\) 是未知信息，即不可控的随机噪声，因此要对 \(u_2\) 求和消去，并且直觉上存在额外噪声导致解码 \(u_1\) 的性能变差；
- In \(W''\), the sole input is \(u_2\), and \(u_1\) has already been decoded; it is known side information available to the decoder before decoding \(u_2\). Mathematically, \(u_1\) is grouped with \(y_1,y_2\) as part of the output. Since \(x_1=u_1\oplus u_2\) and \(u_1\) is known, the decoder can use \(y_1\) to extract additional information about \(u_2\).
  
  而在 \(W''\) 中我们假设 \(u_2\) 是唯一的输入，此时 \(u_1\) 已经被解码，是已知信息，也即 \(u_1\) 作为 decoder 在解码前拿到的辅助信息存在，因此数学上可以和 \(y_1,y_2\) 归在一起；直觉上可以清除 \(x_1=u_1 \oplus u_2\) 对 \(u_2\) 的干扰，从而提升性能。
也即：
\[ W': u_1 \mapsto (y_1,y_2), \text{unknown } u_2 \text{ is marginalized out}; \]

\[ W'': u_2 \mapsto (y_1,y_2,u_1), u_1 \text{ is known side info}. \]
Total information is conserved: \(I(W')+I(W'')=2I(W)\), but redistributed unevenly.

总信息量守恒：\(I(W') + I(W'') = 2I(W)\)，但一多一少。
Recursion. Apply the same transform (copy, XOR, split) recursively to \(W'\) and \(W''\) and we get 4 equivalent channels. After \(n\) levels of recursion, we have \(N=2^n\) sub-channels. Each level makes the good ones better and the bad ones worse.

对 \(W'\) 和 \(W''\) 各自再做同样的 copy-XOR-split，就可以产出 4 个等效信道。\(n\) 层之后有 \(N=2^n\) 个子信道。每一层都让好的更好、差的更差。
Polarization. As \(N\to\infty\), every sub-channel becomes either perfectly reliable (\(I\to 1\)) or completely useless (\(I\to 0\)). The fraction of perfect channels approaches exactly \(I(W)\).

\(N\to\infty\) 时，所有子信道要么 \(I\to 1\)（完美），要么 \(I\to 0\)（无用）。完美的那些占比恰好趋向 \(I(W)\)。
Usage. Transmit data only on the good sub-channels (\(I\approx 1\)). Freeze the bad ones (\(I\approx 0\)) to a fixed known value (e.g., \(0\)). The resulting rate approaches \(I(W)\) i.e. achieves capacity.

只在好的子信道上发数据，坏的 freeze 成 \(0\)。速率 \(\approx I(W)\)，达到容量。

INFOTH Note 20: Channel Coding Schemes

Thu, 13 Nov 2025 01:45:14 +0800

20.1 Maximum Likelihood Decoding 最尤復号法

Note: This is referenced from Toyoaki Nishida’s slides: IT-09.

Also see Atsuyoshi Nakamura’s slides: 情報理論第10回

Setup

Because it’s not so possible for multiple errors to occur, the probability where the output in the output space will be not so far away from the input, if the input space is the same as the output space.

For every message \(i\), we have \(w_i:=e_n(i)=\bold x^n(i)\) defined as the encoded code block or channel input.

INFOTH Note 19: Channel Coding Theorem for DMC 2

Thu, 06 Nov 2025 11:45:14 +0800

19.1 Proof of Channel Coding Theorem for DMC

We inherit the setup in Note 18.

The Setup

DMC: \(p(y|x)\)
Definition of capacity
Encoding and decoding maps, \(((e_n,d_n),n\ge 1)\)
Prob. of error: \(\lambda_{i,n}=\Pr[d_n(\bold y^n)\ne i|\bold x^n=e_n(i)]\)
- Max: \(\lambda_{\max,n}=\max_{i\in[2^{nR}]}\lambda_{i,n}\)
- Average: \(P_e^{(n)}=\frac1{2^{nR}}\sum_{i=1}^{2^{nR}}\lambda_{i,n}\)

Codebook Randomized

Although the channel encoding and decoding maps are fixed, we randomize the encoding map into a codebook where a behavior of ==coding of a message \(m\) into \(\bold x^n\) follows a probability==.

For the \(p(x)\) distribution over \(\mathcal X\), we build a codebook mapping every message \(m\in[2^{nR}=M_n]\) to a sequence of length \(n\). Every symbol of this sequence (\(i\)-th index) is a RV:

INFOTH Note 18: Channel Coding Theorem for DMC 1

Tue, 04 Nov 2025 11:45:14 +0800

18.1 Communication

Setup

There is a sender and a receiver in a communication.

A sender has a message \(m\in[M]:=\{1,2,\dots,M\}\) to send
- The information source codeword (情報源符号): \(m\), inputted into a channel encoder, encoded into \(x\in\mathcal X\)
- Approximately a channel encoder
The sender sends a message via a channel. The channel is noisy, mapping \(x\) to \(y\) probably:
\[ x\to\underbrace{\boxed{p(y|x)}}_W\to y \]
A receiver receives \(y\) and wants to decode \(y\) into the original message \(m\), but turning out to be \(\hat m\):
- The channel outputs \(y\) and the receiver uses a channel decoder to recover \(\hat m\simeq m\)
- Approximately a channel decoder

INFOTH Note 9: Differential Entropy

Thu, 02 Oct 2025 11:45:14 +0800

9.1 Differential Entropy

For discrete r.v.s, we have the entropy. But for real-valued r.v.s with density functions, we need another form of entropy to process things. That is differential entropy.

Def: Diff. Ent.

Let \(X\sim f\) be a real-valued r.v. with density \(f\), meaning

\[ \Pr[X\in[a,b]]=\int_a^b f(x)\mathrm d x,\quad \int_{-\infty}^\infty f(x)\mathrm d x=1. \]

We define \((\int_\R:=\int_{-\infty}^\infty)\)

\[ h(X)=-\int_{\R} f(x)\log f(x)\mathrm d x \]

In fact, the \(h\) may not exist (the integral is \(\pm \infty\)), in case

INFOTH Note 8: BW Transform

Tue, 30 Sep 2025 11:45:14 +0800

8.1 Burrows-Wheeler Transform

Why

BWT is a string (symbol sequence) transform algorithm for data compression. Via rearranging the symbols and making the same symbols more concentrated, it can improve the efficiency of subsequent compression algorithms.

Transformation Algorithm

Consider a sequence of symbols of length \(n\ge 1\) drawn from a finite alphabet \(\mathcal X\) (repetition allowed), call the sequence

\[ x_1x_2\dots x_n \]

Do:

Make a \(n\times n\) array of the cyclic permutations of the sequence like:
\[ x_1x_2\dots x_n\\ x_2x_3\dots x_{n}x_1\\ x_3x_4\dots x_{n}x_1x_2\\ \vdots\\ x_nx_1\dots x_{n-2}x_{n-1} \]
Example: SHANNON original

->
```
SHANNON
HANNONS
ANNONSH
NNONSHA
NONSHAN
ONSHANN
NSHANNO
```

Make every row lexicographically sorted (字典序)

ANNONSH
HANNONS
NNONSHA
NONSHAN
NSHANNO
ONSHANN
SHANNON

As a sender (transformer) we want to send the receiver (de-transformer):
- The last column \(\hat {\bold{x}}\): HSANONN
- The row index of the original sequence \(i\): 7 here starting from 1

INFOTH Note 7: Huffman & Arithmetic Coding

Thu, 25 Sep 2025 11:45:14 +0800

7.1 Huffman Coding

The Algorithm

For a distribution (of appearance) \((p(x),x\in\mathcal X)\),

Sort the probability vector \(\bold p\) in decreasing order,
“Merge” the two smallest probabilities,
Reorder (sort again) and go back to merging.

How to merge: assume we have 2 “symbols” now, say \(a,b\)

Generate a node \(ab\) which has 2 children \(a\) and \(b\)
Assign (e.g.) 0 to \(a\) and \(1\) to \(b\) (to the edges)

Finally, we will get a Huffman tree, where each path from the root (all symbols) to a leaf (one symbol) is the codeword for that symbol.

INFOTH Note 6: Kraft's Inequality, Shannon Code

Tue, 23 Sep 2025 11:45:14 +0800

6.1 Coding and Compression

Def: Code / Encoding

We define a coding/encoding map as follows:

\[ c:\mathcal X^n\to\{0,1\}^*-\empty \]

We say this map is nonsingular iff it’s injective. (thesaurus)

The input is a sequence of source symbols (I call it a source word), and the output (image) is a codeword.

If the < means being a subset of,

Instantaneous codes < Uniquely decodable codes < Nonsingular codes < All codes
Nonsingular: injective (again)
Uniquely decodable: Every sequence concatenated by codewords can only be decoded to only 1 source sequence.
- The “Kleene Closure” of the map is also injective.
- 整个由codeword接成的序列也只能被解码成唯一的source word序列。
If \(A\to0,B\to1,C\to01\), we cannot tell the original seq. from 0101 though this coding is nonsingular.

We can process the coding like a path in a binary tree.

INFOTH Note 5: Fano Ineq., Entropy Rate, AEP

Tue, 16 Sep 2025 11:45:14 +0800

Fano’s Inequality, Markov Chain, Entropy Rate, Asymptotic Equipartition Property and Lossless Compression.

5.1 Fano’s Inequality

Fano is for the scenario of estimation.

Setup

Markov Chain

Given variable \(X\in\cal X\) and \(Y\in\cal Y\). The estimator of \(X\) from \(Y\) is \(\hat X=\hat X(Y)\) (we determine that estimator from \(Y\)).

Note that \(X\to Y\to \hat X\) constructs a Markov Chain, because

\[ P(\hat X=x|Y=y,X=z)=P(\hat X=x|Y=y) \]

Probability of Error

The error as an event: \(\hat X\ne X\)

The probability of error is defined as:

INFOTH Note 4: Cond. MI & Cond. KL, Data Processing Ineq.

Thu, 11 Sep 2025 11:45:14 +0800

4.1 Conditional Mutual Information and Cond. KL Divergence

Def: Cond. MI

\[ I(X;Y|Z):=H(X|Z)-H(X|Y,Z)=H(Y|Z)-H(Y|X,Z) \]

展开式子可以发现，在互信息中定义的 给定 Z 也是 将 Z 求和的：

\[ \begin{aligned} I(X;Y|Z)&=H(X|Z)-H(X|Y,Z) \\&=\sum_z p(z)\sum_x p(x|z)\log \frac1{p(x|z)} \\&+\sum_{z}p(z)\sum_y p(y|z)\sum_x p(x|y,z)\log p(x|y,z)\quad\text{(Def)} \\&=\sum_{x,z}p(x,z)\log \frac1{p(x|z)}+ \sum_{x,y,z}p(x,y,z)\log{p(x|y,z)} \\&=\sum_{x,y,z}p(x,y,z)\log \frac1{p(x|z)}+ \sum_{x,y,z}p(x,y,z)\log{p(x|y,z)} \\&=\sum_{x,y,z}p(x,y,z)\log {p(x|y,z)\over p(x|z)} \\&=\sum_{x,y,z}p(x,y,z)\log {p(x,y|z)\over p(x|z)p(y|z)} \\&=E_{X,Y,Z}\left[\log {p(x,y|z)\over p(x|z)p(y|z)}\right] \end{aligned} \]

We can also write the expectation as

INFOTH Note 3: Relative Ent. (KL), Mutual Info.

Thu, 04 Sep 2025 11:45:14 +0800

I now start cancel numbering for sections since I’m lazy.

3.1 Log-sum Inequality

Theorem

Suppose \(a_i,b_i>0\) for a countable set \(\mathcal X\), then

\[ \sum_{i\in\mathcal X}a_i\log {a_i\over b_i}\ge a\log b,\quad a=\sum_i a_i, \quad b=\sum_i b_i. \]

Proof

Using Jensen’s Inequality, given \(u\log u\) is convex and \(\sum_i (b_i/b) = 1\)

\[ \begin{aligned} \sum_{i\in\cal X}a_i\log {a_i\over b_i}&=b\sum_{i}({b_i\over b}){a_i\over b_i}\log {a_i\over b_i} \\&= b\sum_i \lambda_i u_i\log u_i,\qquad\lambda_i:=b_i/b,\quad\sum_i\lambda_i=1,\quad u_i=a_i/b_i, \\&\ge b\cdot (\sum_i\lambda_iu_i) \cdot \log \sum_i\lambda_iu_i \\&=b\cdot(\sum_i{a_i\over b})\cdot\log \sum_i {a_i\over b} \\&=a\log{a\over b}.\qquad\square \end{aligned} \]

3.2 Relative Entropy (KL distance)

Def: Relative Entropy 相对熵 = KL散度

The relative entropy i.e. Kullback-Leibler (KL) Distance between 2 probability mass functions \(p(x)\) and \(q(x)\) is defined as

INFOTH Note 2: Joint & Conditional Entropy, Chain Rule

Tue, 02 Sep 2025 23:39:54 +0800

2.1 Entropy

Def 2.1.1 Entropy

We define, for a discrete r.v. \(X\) on a domain \(\mathcal X\) with finitely many or countably infinitely many elements (such as a finite set or \(\Z\)), a distribution \(p\in\Delta_{\cal X}\):

\[ H(X)=-\sum_{x\in\mathcal X}p(x)\log p(x)\equiv E_{X\sim p} \log {1\over p(X)} \]

\(E_p\) 在这里指的是 \(X\) under \(p\) distribution的时候，\(X\) 对某个函数的期望。This specifies an equivalent definition by expectation of entropy.

Properties

\(H(X)\ge 0\)
Replacing the base 替换底数：Let \(H_a(X)\) be the entropy defined in the base of the log changed into \(a\) instead of 2.

\(H_b(X)=(\log_b a)H_a(X)\)
- So \(H_a(X)=(\log_2a )H(X)\).

INFOTH Note 1: Axioms & Def of Entropy

Thu, 28 Aug 2025 23:39:41 +0800

1.1 Definition of the Shannon Entropy

To (re)invent the entropy, we need to let this quantity follow some axioms. Let the entropy be \(H(\cdot)\).

\(H\) can also wrap distribution like \(p_1,...,p_n\), or just be fed with a distribution tuple \(p\), or a random variable \(X\).

Def 1.1.1 Shannon Entropy

Consider a discrete probability distribution \(p=(p_1,\dots,p_n)\) on the set \(\mathcal X=\{1,\dots,n\}\).

The Shannon Entropy of \(p\) is defined by

\[ H(p)=-\sum_{x\in\cal X}p_x\log p_x. \]

\(\log:=\log_2\), as this is Shannon’s convention (and also will be ours)
We have defined the entropy only on a finite set.
We also allow \(H\) to wrap a tuple like this: \(H(p)=H(p_1,\dots,p_n)\).

Def 1.1.2 Entropy of a Random Variable

Given a discrete r.v. \(X\) over a finitely many set \(\cal X\), we write the entropy as