What would you like to understand?

Conceptually, this formula identifies mutual information as the Kullback-Leibler (KL) divergence between the actual joint distribution $p(x,y)$ and the independent reference distribution $p(x)p(y)$. It measures how far the actual joint distribution is from pure independence.

Conditions: The displayed derivation is for discrete distributions using probability mass functions.; The ratio form requires $p(x,y)>0$ and $p(x)p(y)>0$ on the summed support.

Conceptually, this formula identifies mutual information as the Kullback-Leibler (KL) divergence between the actual joint distribution $p(x,y)$ and the independent reference distribution $p(x)p(y)$. It measures how far the actual joint distribution is from pure independence.

Conditions: The displayed derivation is for discrete distributions using probability mass functions.; The ratio form requires $p(x,y)>0$ and $p(x)p(y)>0$ on the summed support.