Why is mutual information equal to $H(X)$ minus H(X|Y)?

Conditional entropy H(X|Y) measures the average remaining uncertainty or surprise about a random variable X after another random variable Y has been observed. It is defined as the expected value of the conditional self-information h(x|y) over the joint distribution of X and Y.

Conditions: X and Y are discrete random variables.; The sum is taken over the joint alphabet $A_x \times A_y$.

Joint entropy $H(X,Y)$ is not equal to $H(X) + H(Y)$ because adding the marginal entropies double-counts the mutual information I(X;Y) shared between the variables. The correct relationship is $H(X,Y) = H(X) + H(Y) - I$(X;Y).

Conditions: X and Y are random variables that may share information (i.e., they are not necessarily independent).

In the entropy Venn diagram, the left circle represents the entropy $H(X)$ and the right circle represents $H(Y)$. Their intersection represents the mutual information $I(X;Y)$.

Conditions: Suitable for illustrating information decomposition between two discrete random variables $X$ and $Y$.

When $X$ and $Y$ are independent, knowing one tells you nothing about the other. Mathematically, the joint distribution factors into the product of the marginals, $p(x,y) = p(x) \cdot p(y)$.

Conditions: Applies to the case where $X$ and $Y$ are independent.; The displayed notation uses probability mass functions $p(x)$, $p(y)$, and $p(x,y)$.

Yes, mutual information is perfectly symmetric. The formula $I(X;Y) = I(Y;X)$ holds true.

Conditions: Applies to any pair of random variables $X$ and $Y$ for which mutual information is defined.