Relative entropy

KL divergence compares distributions and is generally asymmetric, with support conditions affecting finiteness.

Different ways to understand it

No reviewed video yet. Explore the concept in a map or suggest a resource. Suggest a video

Mutual information
Mutual information measures statistical dependence as the reduction in uncertainty from observing another variable.
Shannon entropy
Entropy is expected negative log probability for a discrete distribution.
Channel capacity
Capacity bounds achievable reliable information rates for a specified channel model.
Conditional entropy
Conditional entropy averages remaining uncertainty after another variable is observed.
Error correction bounds
Minimum distance d permits correcting up to floor((d-1)/2) arbitrary symbol errors.