When does conditional entropy H(X|Y) equal unconditional entropy ?
Conditions
- X and Y are discrete random variables.
Reasoning, step by step
- Recall the bound (X|Y) ≤ .
- Identify the upper bound case where H(X|Y) = .
- Recognize that this equality holds precisely when the conditional distribution p(x|y) is equal to the marginal distribution for all y.
- Conclude that this condition defines statistical independence between X and Y.
Example
The lecturer states: 'we may find that the conditional entropy for X given Y is the same as the unconditioned entropy of X, and that is the case if our variables X and Y are independent'.
Common misconceptions
- Assuming that H(X|Y) = implies a strong correlation between X and Y.
- Thinking that independence means H(X|Y) = 0, whereas independence actually means conditioning provides no reduction in entropy.
Watch the explanation
Connected concepts
Explore next
Related questions
Conditional entropy H(X|Y) measures the average remaining uncertainty or surprise about a random variable X after another random variable Y has been observed. It is defined as the expected value of the conditional self-information h(x|y) over the joint distribution of X and Y.
Conditions: X and Y are discrete random variables.; The sum is taken over the joint alphabet .
Joint entropy is not equal to because adding the marginal entropies double-counts the mutual information I(X;Y) shared between the variables. The correct relationship is (X;Y).
Conditions: X and Y are random variables that may share information (i.e., they are not necessarily independent).
When conditional entropy H(X|Y) is zero, it means there is no remaining uncertainty or surprise about X once Y is known. This implies that X is a deterministic function of Y, so all information about X is completely contained within Y.
Conditions: X and Y are discrete random variables.
Conditional entropy is the difference between the joint entropy of X and Y and the marginal entropy of Y. This relationship is expressed by the entropy chain rule: H(X|Y) = .
Conditions: X and Y are discrete random variables with well-defined joint and marginal entropies.
Conditional entropy H(X|Y) is bounded by (X|Y) ≤ because learning the value of Y provides information that can only reduce or leave unchanged the average uncertainty about X; it cannot increase it on average.
Conditions: X and Y are discrete random variables.
Answers are generated from source material and independently checked. Consult the original video or creator if something is unclear.