Why is conditional entropy H(X|Y) always less than or equal to unconditional entropy ?
Conditions
- X and Y are discrete random variables.
Reasoning, step by step
- Understand that represents the total uncertainty in X before observing anything else.
- Recognize that observing Y reduces the sample space or probability distribution of X to p(x|y).
- Since information cannot create uncertainty on average, the remaining uncertainty H(X|Y) must be less than or equal to the original uncertainty .
- The lower bound is 0, which occurs when Y completely determines X.
Example
The lecturer states: 'the average conditional entropy is always less than or equal to the unconditional entropy... it does not increase the uncertainty I should say on average. It may leave the uncertainty the same, but it's not going to increase it on average.'
Common misconceptions
- Thinking that learning additional information Y might increase uncertainty about X.
- Confusing the average behavior (entropy) with pointwise behavior (self-information), where conditioning can sometimes increase surprise for specific outcomes.
Watch the explanation
Connected concepts
Explore next
Related questions
Conditional entropy H(X|Y) measures the average remaining uncertainty or surprise about a random variable X after another random variable Y has been observed. It is defined as the expected value of the conditional self-information h(x|y) over the joint distribution of X and Y.
Conditions: X and Y are discrete random variables.; The sum is taken over the joint alphabet .
Joint entropy is not equal to because adding the marginal entropies double-counts the mutual information I(X;Y) shared between the variables. The correct relationship is (X;Y).
Conditions: X and Y are random variables that may share information (i.e., they are not necessarily independent).
When conditional entropy H(X|Y) is zero, it means there is no remaining uncertainty or surprise about X once Y is known. This implies that X is a deterministic function of Y, so all information about X is completely contained within Y.
Conditions: X and Y are discrete random variables.
Conditional entropy is the difference between the joint entropy of X and Y and the marginal entropy of Y. This relationship is expressed by the entropy chain rule: H(X|Y) = .
Conditions: X and Y are discrete random variables with well-defined joint and marginal entropies.
Conditional entropy H(X|Y) equals unconditional entropy if and only if the random variables X and Y are statistically independent. In this case, knowing Y provides no information about X, so the uncertainty remains unchanged.
Conditions: X and Y are discrete random variables.
Answers are generated from source material and independently checked. Consult the original video or creator if something is unclear.