What is conditional entropy H(X|Y) and how is it defined for discrete random variables?
Conditions
- X and Y are discrete random variables.
- The sum is taken over the joint alphabet .
Reasoning, step by step
- Define the conditional self-information for specific outcomes x and y as h(x|y) = -\log_2 p(x|y).
- Take the expected value of h(x|y) over the joint probability distribution .
- This yields the formula H(X|Y) = .
Example
The slide displays the formula H(X|Y) = and defines it as the average surprise remaining about sample x of X if we already know the sample y of Y.
Common misconceptions
- Confusing conditional entropy with joint entropy or marginal entropy.
- Assuming H(X|Y) is the entropy of X minus the entropy of Y, rather than an expectation over the joint distribution.
Watch the explanation
Connected concepts
Explore next
Related questions
Joint entropy is not equal to because adding the marginal entropies double-counts the mutual information I(X;Y) shared between the variables. The correct relationship is (X;Y).
Conditions: X and Y are random variables that may share information (i.e., they are not necessarily independent).
When conditional entropy H(X|Y) is zero, it means there is no remaining uncertainty or surprise about X once Y is known. This implies that X is a deterministic function of Y, so all information about X is completely contained within Y.
Conditions: X and Y are discrete random variables.
Conditional entropy is the difference between the joint entropy of X and Y and the marginal entropy of Y. This relationship is expressed by the entropy chain rule: H(X|Y) = .
Conditions: X and Y are discrete random variables with well-defined joint and marginal entropies.
Conditional entropy H(X|Y) is bounded by (X|Y) ≤ because learning the value of Y provides information that can only reduce or leave unchanged the average uncertainty about X; it cannot increase it on average.
Conditions: X and Y are discrete random variables.
Conditional entropy H(X|Y) equals unconditional entropy if and only if the random variables X and Y are statistically independent. In this case, knowing Y provides no information about X, so the uncertainty remains unchanged.
Conditions: X and Y are discrete random variables.
Answers are generated from source material and independently checked. Consult the original video or creator if something is unclear.