Shannon entropy

Entropy is expected negative log probability for a discrete distribution.

H(X)=-\sum_x p(x)\log p(x)

Different ways to understand it

No reviewed video yet. Explore the concept in a map or suggest a resource. Suggest a video

Conditional entropy
Conditional entropy averages remaining uncertainty after another variable is observed.
Probability
A probability measure assigns nonnegative weights to events, with total weight one and countable additivity on disjoint events.
Relative entropy
KL divergence compares distributions and is generally asymmetric, with support conditions affecting finiteness.
Source coding
Source coding represents messages efficiently, with entropy constraining lossless average code length.
Channel capacity
Capacity bounds achievable reliable information rates for a specified channel model.