Introduction to Information Theory: Entropy - Part 6 - Conditional Entropy

1. Formula
   Observation

   The slide displays the capital letter X in expressions such as H(X|Y) and in the text 'sample x of X'.

X

A discrete random variable whose uncertainty is being considered.

Discrete random variable

1. Formula
   Observation

   The slide displays the capital letter Y in expressions such as H(X|Y) and $h(y)$.

Y

Another discrete random variable, known or observed, which provides information about X.

Discrete random variable

1. Formula
   Observation

   The slide uses lowercase x in h(x|y) and in the summation index $x \in A_x$.

x

A specific realization or sample value of the random variable X.

Realization of X

1. Formula
   Observation

   The slide uses lowercase y in h(x|y) and in the summation index $y \in A_y$.

y

A specific realization or sample value of the random variable Y.

Realization of Y

1. Formula
   Observation

   The slide defines h(x|y) = -\log_2 p(x|y).

h

Self-information (or surprisal) function; lowercase denotes a specific realization.

Function returning a scalar value

1. Formula
   Observation

   The slide defines H(X|Y) as an average over joint probabilities.

H

Entropy function; uppercase denotes the expected value (average) over a random variable.

Function returning a scalar value

1. Formula
   Observation

   The slide uses $p(x,y)$ and p(x|y) in the formulas.

p

Probability mass function; $p(x,y)$ is the joint probability, p(x|y) is the conditional probability.

Probability function

1. Formula
   Observation

   The slide shows summations over $x \in A_x$ and $y \in A_y$.

1. The video does not explicitly define the sets $A_x$ and $A_y$, though they represent the alphabets or support of the variables.

$A_x$, $A_y$

The alphabets (sets of possible values) for the random variables X and Y, respectively.

Sets of discrete values

1. Formula
   Observation

   The slide displays the formula h(x|y) = -log_2 p(x|y).

h(x|y)

Conditional Shannon information content (surprise) for a specific sample x of X given a specific sample y of Y.

Real numbers, non-negative.

1. Formula
   Observation

   The slide displays the formula H(X|Y) = $\sum_{x \in A_x} \sum_{y \in A_y} p(x,y) \log_2 \frac{1}{p(x|y)}$.

H(X|Y)

Average conditional entropy of random variable X given random variable Y.

Real numbers, non-negative.

1. Formula
   Observation

   Appears in the denominator of the logarithm in the conditional entropy formula: p(x|y).

p(x|y)

Conditional probability of event x given event y.

Interval [0, 1].

1. Formula
   Observation

   Appears as the weighting factor in the double summation: $p(x,y)$.

$p(x,y)$

Joint probability of events x and y occurring together.

Interval [0, 1].

1. Formula
   Observation

   The slide displays the formula h(x|y) = -\log_2 p(x|y).

The conditional surprisal (or conditional self-information) measures the specific surprise of observing a realization x of variable X, given that we already know the realization y of variable Y. It is defined using the conditional probability p(x|y).

1. p(x|y) > 0

1. Joint Surprisal

1. Formula
   Observation

   The slide displays the formula H(X|Y) = $\sum_{x \in A_x} \sum_{y \in A_y} p(x,y) \log_2 \frac{1}{p(x|y)}$.

2. Audio
   Observation

   The speaker explains it as the average uncertainty remaining about X given what we already know about Y.

Conditional entropy represents the average uncertainty remaining about a random variable X after another random variable Y has been observed. It is the expected value of the conditional surprisal over the joint distribution of X and Y.

1. X and Y are discrete random variables

1. Conditional Surprisal
2. Joint Entropy

1. Formula
   Observation

   The slide displays the formula $h(x,y)$ implicitly via the relation h(x|y) = $h(x,y) - h(y)$.

1. The explicit definition $h(x,y) = -\log_2 p(x,y)$ is not written on the slide, but is referenced in the identity.

The joint surprisal measures the specific surprise of observing the pair of realizations (x, y) together. It is related to the conditional surprisal and marginal surprisal.

1. $p(x,y) > 0$

1. Formula
   Observation

   The slide displays $H(X,Y)$ in the relation H(X|Y) = $H(X,Y) - H(Y)$.

1. The explicit summation formula for $H(X,Y)$ is not shown, only its use in the identity.

Joint entropy is the average uncertainty associated with a pair of random variables (X, Y) considered together.

1. X and Y are discrete random variables

1. Formula
   Observation

   The slide displays $H(Y)$ in the relation H(X|Y) = $H(X,Y) - H(Y)$.

2. Audio
   Observation

   The speaker refers to $H(Y)$ as the 'marginal entropy' of variable Y.

Marginal entropy is the entropy of a single random variable within a joint setting, calculated using its marginal probability distribution.

1. Y is a discrete random variable

1. Formula
   Observation

   Slide text defines Conditional entropy as '(average) surprise remaining about sample x of X if we already know the sample y of Y'.

2. Audio
   Observation

   Speaker explains changing to look at the conditional probability of X given Y and averaging over the joint space.

Conditional entropy measures the average uncertainty or surprise remaining about a random variable X after observing another random variable Y. It is calculated by taking the expected value of the conditional self-information over the joint distribution of X and Y.

1. X and Y are discrete random variables.

2. The sum is taken over the joint alphabet $A_x \times A_y$.

1. Formula
   Observation

   Slide displays H(X|Y) = $H(X,Y) - H(Y)$.

The conditional entropy of X given Y can be expressed as the difference between the joint entropy of X and Y and the marginal entropy of Y.

1. Applies to discrete random variables X and Y.

1. Definition of Conditional Entropy

1. Audio
   Observation

   Speaker states 'We can draw Venn diagrams to represent the uncertainty or the information content in each variable.'

2. Animation
   Observation

   Red and blue circles appear and overlap to form a Venn diagram representing $H(X)$, $H(Y)$, $H(X,Y)$, I(X;Y), H(X|Y), and H(Y|X).

Entropy quantities can be visualized using a Venn diagram where the area of a circle represents the marginal entropy of a variable. The union of circles represents joint entropy, the intersection represents mutual information, and the non-overlapping crescents represent conditional entropies.

1. Visual analogy only; areas correspond to information quantities in bits.

1. Definition of Conditional Entropy

1. Formula
   Observation

   h(x|y) = $h(x,y) - h(y)$

2. Formula
   Observation

   H(X|Y) = $H(X,Y) - H(Y)$

Conditional entropy represents the average surprise remaining about a sample x of random variable X if we already know the sample y of random variable Y. It is defined as the difference between joint entropy and the entropy of the conditioning variable.

1. X and Y are discrete random variables

2. $p(x,y) > 0$ for all pairs in the summation

1. Audio
   Observation

   the average conditional entropy is always less than or equal to the unconditional entropy

2. Formula
   Observation

   $0 \le H$(X|Y) $\le H(X)$

The conditional entropy is bounded between zero and the unconditional entropy. Learning the value of Y reduces (or leaves unchanged) the uncertainty about X on average, but cannot increase it.

1. X and Y are discrete random variables

1. Definition of Conditional Entropy

1. Audio
   Observation

   we may find that the conditional entropy for X given Y is the same as the unconditioned entropy of X, and that is the case if our variables X and Y are independent

2. Formula
   Observation

   H(X|Y) = $H(X) \text{ iff } X \text{ and } Y \text{ are independent}$

Conditional entropy equals unconditional entropy if and only if the two random variables are statistically independent, meaning knowing Y provides no information about X.

1. X and Y are discrete random variables

1. Definition of Conditional Entropy

1. Audio
   Observation

   our conditional entropy being zero simply means there is no surprise left in X once we know Y

2. Formula
   Observation

   H(X|Y) = 0

When conditional entropy is zero, all information about X is contained within Y; knowing Y completely determines X with no remaining uncertainty.

1. X and Y are discrete random variables

1. Definition of Conditional Entropy

1. Formula
   Observation

   The slide explicitly states h(x|y) = $h(x,y) - h(y)$.

The conditional surprisal of x given y is equal to the joint surprisal of x and y minus the marginal surprisal of y.

1. x and y are realizations of discrete random variables X and Y

For all valid realizations x and y.

1. Formula
   Observation

   The slide explicitly states H(X|Y) = $H(X,Y) - H(Y)$.

2. Audio
   Observation

   The speaker explains this as the difference between the joint entropy and the marginal entropy.

The conditional entropy of X given Y is equal to the joint entropy of X and Y minus the marginal entropy of Y.

1. X and Y are discrete random variables

For all discrete random variables X and Y.

1. Audio
   Observation

   Speaker says 'we're not adding the two circles because that would double count this overlapping section... joint entropy is not necessarily as large as the separate sum of marginal entropies'.

2. Diagram
   Observation

   Arrows point to three distinct regions (left crescent, intersection, right crescent) summing to the total union area labeled $H(X,Y)$.

The joint entropy $H(X,Y)$ is equal to the sum of the three disjoint areas in the Venn diagram (H(X|Y) + I(X;Y) + H(Y|X)), which is less than or equal to the sum of the marginal entropies $H(X) + H(Y)$. Equality holds if and only if X and Y are independent (intersection is zero).

1. X and Y are random variables represented by overlapping sets.

For all random variables X and Y.

1. Audio
   Observation

   This is a very classic diagrammatic way to represent entropies and conditional entropies.

2. Diagram
   Observation

   Venn diagram showing overlapping circles for $H(X)$ and $H(Y)$ with intersection labeled $H(X,Y)$ and non-overlapping regions labeled H(X|Y) and H(Y|X)

Entropies and conditional entropies can be represented using a Venn diagram where the total area corresponds to joint entropy, individual circle areas correspond to marginal entropies, and the non-overlapping parts correspond to conditional entropies.

1. X and Y are discrete random variables

For all pairs of discrete random variables X and Y

1. Audio
   Observation

   Lecturer says: 'if we see Q, we are fairly certain that the next letter ... is going to be U' and 'we have very low surprise in encountering a U next'.

1. This is an intuitive linguistic example rather than a formal theorem.

In English text, knowing that the previous character is Q makes the next character U highly predictable, so the conditional entropy of the next character given Q is very low.

1. The source is English text.

2. The previous character is Q.

For the illustrative English-text example, conditional on seeing Q, the next character is strongly expected to be U.

1. Audio
   Observation

   Lecturer states that conditional entropy of one character given the character beforehand is generally lower than the entropy of the character otherwise.

1. The lecturer says 'generally', not a universally quantified theorem, and no counterexample discussion is provided in this clip.

For correlated sources such as English text, the conditional entropy of a symbol given preceding context is typically smaller than its unconditional entropy.

1. There is statistical dependence between the current symbol and its context.

General tendency asserted for the discussed source model, not proved in the clip.

1. Formula
   Observation

   The slide presents the sequence of equations leading to H(X|Y) = $H(X,Y) - H(Y)$.

1. The intermediate algebraic steps showing why h(x|y) = $h(x,y) - h(y)$ implies the entropy relation are not explicitly written out on the slide, though the result is stated.

1. Expression

   $$h(x|y) = h(x,y) - h(y)$$
   Explanation

   Start with the definition of conditional surprisal as the difference between joint and marginal surprisal.

   Justification

   Definition of conditional surprisal.

   Shown in the video
2. Expression

   $$H(X|Y) = E[h(X|Y)]$$
   Explanation

   Take the expected value of both sides over the joint distribution of X and Y.

   Justification

   Definition of entropy as expected surprisal.

   Derived from the video
3. Expression

   $$E[h(X,Y) - h(Y)] = E[h(X,Y)] - E[h(Y)]$$
   Explanation

   By linearity of expectation, the expected value of the difference is the difference of the expected values.

   Justification

   Linearity of expectation.

   Derived from the video
4. Expression

   $$H(X|Y) = H(X,Y) - H(Y)$$
   Explanation

   Substitute the definitions of joint and marginal entropy back into the equation.

   Justification

   Definitions of $H(X,Y)$ and $H(Y)$.

   Shown in the video

The conditional entropy H(X|Y) can be computed as the joint entropy $H(X,Y)$ minus the marginal entropy $H(Y)$.

1. Animation
   Observation

   Labels $H(X,Y)$, I(X;Y), H(X|Y), and H(Y|X) appear sequentially pointing to specific regions of the Venn diagram.

2. Audio
   Observation

   Speaker explains that joint entropy is the sum of three separate areas without double counting the overlap.

1. Expression

   $$H(X) = \text{Area}(\text{Red Circle}) = \text{Left Crescent} + \text{Intersection}$$
   Explanation

   The total entropy of X comprises the unique uncertainty and the shared uncertainty with Y.

   Justification

   Definition of marginal entropy in the visual model.

   Derived from the video
2. Expression

   $$H(Y) = \text{Area}(\text{Blue Circle}) = \text{Right Crescent} + \text{Intersection}$$
   Explanation

   The total entropy of Y comprises its unique uncertainty and the shared uncertainty with X.

   Justification

   Definition of marginal entropy in the visual model.

   Derived from the video
3. Expression

   $$I(X;Y) = \text{Intersection}$$
   Explanation

   The overlapping region represents the information shared between X and Y.

   Justification

   Definition of mutual information in the visual model.

   Shown in the video
4. Expression

   $$H(X|Y) = \text{Left Crescent} = H(X) - I(X;Y)$$
   Explanation

   The remaining uncertainty in X after knowing Y is the part of X's circle not overlapping with Y.

   Justification

   Visual subtraction of the intersection from the red circle.

   Shown in the video
5. Expression

   $$H(X,Y) = \text{Union} = \text{Left Crescent} + \text{Intersection} + \text{Right Crescent}$$
   Explanation

   The joint entropy is the total area covered by either circle, ensuring the intersection is counted only once.

   Justification

   Set theory union principle applied to information measures.

   Shown in the video

The Venn diagram provides a consistent geometric interpretation of the relationships $H(X,Y) = H(X) + H$(Y|X) = $H(Y) + H$(X|Y) = I(X;Y) + H(X|Y) + H(Y|X).

1. Formula
   Observation

   H(X|Y) = $H(X,Y) - H(Y)$

1. Expression

   $$H(X,Y) = H(X) + H(Y|X)$$
   Explanation

   Joint entropy decomposes into marginal entropy plus conditional entropy

   Justification

   Definition of joint entropy

   Shown in the video
2. Expression

   $$H(X,Y) = H(Y) + H(X|Y)$$
   Explanation

   Symmetric decomposition of joint entropy

   Justification

   Definition of joint entropy

   Shown in the video
3. Expression

   $$H(X|Y) = H(X,Y) - H(Y)$$
   Explanation

   Rearranging gives conditional entropy as difference

   Justification

   Algebraic manipulation

   Shown in the video

Conditional entropy equals joint entropy minus the entropy of the conditioning variable

1. Diagram
   Observation

   Venn diagram labels $H(X,Y)$ in the overlap, $H(Y)$ in the right circle, H(X|Y) in the left-only region, and $H(X)$ in the left circle.

2. Formula
   Observation

   Slide simultaneously displays H(X|Y)=$H(X,Y)-H(Y)$.

1. The clip gives the identity and visual analogy but does not present an algebraic derivation from expectations.

1. Expression

   $$H(X,Y)$$
   Explanation

   The overlap region is identified as the joint entropy of X and Y.

   Justification

   Directly labelled in the Venn diagram.

   Shown in the video
2. Expression

   $$H(Y)$$
   Explanation

   The right circle is identified as the entropy of Y alone.

   Justification

   Directly labelled in the Venn diagram.

   Shown in the video
3. Expression

   $$H(X|Y)$$
   Explanation

   The portion of the left circle outside the overlap is identified as the conditional entropy of X given Y.

   Justification

   Directly labelled in the Venn diagram.

   Shown in the video
4. Expression

   $$H(X|Y) = H(X,Y) - H(Y)$$
   Explanation

   The slide states the subtraction relation corresponding to the pictured decomposition.

   Justification

   Formula printed on the same slide as the diagram.

   Shown in the video

The diagram and formula together present conditional entropy as the part of the joint uncertainty not already accounted for by Y.

1. Audio
   Observation

   Just take an example here coming back to looking at coding characters in English text.

2. Diagram
   Observation

   Table 2.9 showing Shannon information contents of English letters with probabilities and self-information values

1. Specific variable Y that would reduce H(X|Y) is posed as a question but not answered in this clip

What variable Y would reduce the uncertainty of character X in English text, thereby decreasing code length for conditional encoding?

1. X represents a character in English text

2. Shannon information contents table provided for English letters

Identify a conditioning variable that reduces conditional entropy and enables shorter codes

1. Expression

   $$$$
   Explanation

   Consider what prior information about English text could reduce uncertainty about the next character

   Justification

   Contextual knowledge in natural language

   Shown in the video
2. Expression

   $$$$
   Explanation

   If receiver knows previous characters, conditional entropy H(X|Y) < $H(X)$, allowing shorter codes

   Justification

   Property of conditional entropy bounds

   Derived from the video

The specific variable Y is not identified in this segment; the question is posed for further exploration

Conceptual verification through the bounds property: H(X|Y) ≤ $H(X)$

1. Caption evidence
   Observation

   Slide heading 'Example 1:' followed by coding characters in English text and asking what variable Y would drop H(X|Y) and therefore the code length for a conditional encoding of incoming character X.

2. Audio
   Observation

   Lecturer discusses coding letters given the previous letter and uses the Q->U example.

3. Diagram
   Observation

   Right side shows Table 2.9 Shannon information contents of the consonant n with letter frequencies and $h(p_i)$ values.

1. The table is visible but not read aloud in detail during this clip.

2. The lecturer does not compute a numerical value for H(X|Y) on-screen in this segment.

Use conditional entropy to explain why encoding an incoming English character can require fewer bits if the previous character is known.

1. X denotes the incoming character.

2. Y denotes the previous character or context.

3. English letter statistics make some transitions highly predictable, e.g. Q is often followed by U.

Explain how conditioning on context lowers remaining surprise and therefore code length.

1. Expression

   $$Y = previous character$$
   Explanation

   Choose the prior letter as the conditioning variable.

   Justification

   Stated in the slide example and in the lecturer's explanation.

   Shown in the video
2. Expression

   $$H(X|Y)$$
   Explanation

   Measure the remaining uncertainty in the next letter after observing the previous one.

   Justification

   This is the quantity defined on the slide.

   Shown in the video
3. Expression

   $$Q \Rightarrow U$$
   Explanation

   If the previous letter is Q, the next letter U is highly expected.

   Justification

   Explicit spoken example from the lecturer.

   Shown in the video
4. Expression

   $$H(X|Y=Q) \text{ is very low}$$
   Explanation

   Because U is nearly determined after Q, the conditional uncertainty is small.

   Justification

   Spoken interpretation by the lecturer.

   Shown in the video
5. Expression

   $$lower H(X|Y) \Rightarrow shorter code length$$
   Explanation

   Reduced conditional entropy translates into fewer bits on average for a conditional encoder.

   Justification

   Slide text links H(X|Y) to code length for conditional encoding.

   Shown in the video

Conditioning on previous context such as Q reduces the entropy of the next character, so a conditional code can use fewer bits than an unconditional code.

The lecturer verifies the idea intuitively through the near-certainty of U following Q and by connecting lower conditional entropy to shorter code length.

1. Caption evidence
   Observation

   Exercise slide lists tests: conditionalentropy([0.5,0;0,0.5])=0 and conditionalentropy([0.25,0.25;0.25,0.25])=1.

1. No worked computation is shown in the clip; only the expected results are printed.

2. The reasoning behind these exact numeric outputs is not verbalized here.

Check the behaviour of the proposed Matlab function conditionalentropy(p) on simple joint distributions.

1. p is a 2D joint probability matrix $p(x,y)$.

2. Test 1: p=[0.5,0;0,0.5].

3. Test 2: p=[0.25,0.25;0.25,0.25].

Identify the stated outputs of conditionalentropy(p) for these matrices.

1. Expression

   $$conditionalentropy([0.5,0;0,0.5]) = 0$$
   Explanation

   The slide states that this deterministic diagonal joint distribution yields zero conditional entropy.

   Justification

   Printed test case on the exercise slide.

   Shown in the video
2. Expression

   $$conditionalentropy([0.25,0.25;0.25,0.25]) = 1$$
   Explanation

   The slide states that the uniform 2x2 joint distribution yields conditional entropy 1.

   Justification

   Printed test case on the exercise slide.

   Shown in the video

The stated outputs are 0 for [0.5,0;0,0.5] and 1 for [0.25,0.25;0.25,0.25].

Verification is only by the printed expected results; the clip does not show the derivation.

1. Diagram
   Observation

   A static presentation slide titled 'Conditional entropy' is displayed throughout the clip, containing text and mathematical formulas.

1. Title text

2. Bullet points

3. Mathematical formulas

1. The content of the slide remains static and unchanged for the entire duration of the clip.

The visual serves as a reference for the definitions and relationships being explained by the speaker's audio narration.

1. Animation
   Observation

   A solid red circle appears on the left side of the screen with the label $H(X)$ below it.

1. Red circle

2. Label $H(X)$

1. Circle fades in

2. Label appears

1. Color red

2. Position left

Represents the total uncertainty or information content of random variable X.

1. Animation
   Observation

   A semi-transparent blue circle appears on the right, overlapping the red circle. Labels $H(Y)$ and '+' signs appear in the non-overlapping regions.

1. Blue circle

2. Label $H(Y)$

3. Plus signs

1. Blue circle fades in

2. Overlap region becomes purple/darker

3. Plus signs indicate additive components

1. Red circle remains

2. Positions fixed

Introduces the second variable Y and visually establishes that general variables share some information (overlap) and have some unique information (crescents).

1. Animation
   Observation

   Labels $H(X,Y)$ and I(X;Y) appear with arrows pointing to the union and intersection respectively.

1. Label $H(X,Y)$

2. Label I(X;Y)

3. Arrows

1. Arrows draw attention to specific regions

2. Text labels appear

1. Circles unchanged

Identifies the total combined uncertainty (union) and the shared information (intersection).

1. Animation
   Observation

   Labels H(X|Y) and H(Y|X) appear with arrows pointing to the left and right crescents respectively.

1. Label H(X|Y)

2. Label H(Y|X)

3. Arrows

1. Arrows point to non-overlapping regions

1. Previous labels remain

Visually defines conditional entropy as the residual uncertainty in one variable after accounting for the information provided by the other (removing the intersection).

1. Diagram
   Observation

   Venn diagram with red circle $H(X)$, blue circle $H(Y)$, intersection $H(X,Y)$, left crescent H(X|Y), right crescent H(Y|X)

1. Red circle representing $H(X)$

2. Blue circle representing $H(Y)$

3. Intersection region $H(X,Y)$

4. Left non-overlapping region H(X|Y)

5. Right non-overlapping region H(Y|X)

1. Diagram remains static throughout the segment

1. Relative sizes of regions reflect entropy relationships

2. Labels remain constant

Visual representation showing how joint entropy decomposes into marginal and conditional components, with overlap representing shared information

1. Animation
   Observation

   Slide transitions from Page 29 to Page 30 showing new content

1. Previous slide with definitions and Venn diagram

2. New slide with example and probability table

1. Content shifts from theoretical properties to practical example

2. Probability table for English letters appears

1. Conditional entropy definition remains visible

2. Venn diagram persists

Transition marks shift from abstract properties to concrete application in text coding

1. Diagram
   Observation

   Two overlapping circles labelled $H(X)$ and $H(Y)$; overlap labelled $H(X,Y)$; left-only region labelled H(X|Y); arrows indicate addition/subtraction among regions.

1. The exact arrow semantics are schematic rather than rigorously annotated.

1. Left circle labelled $H(X)$

2. Right circle labelled $H(Y)$

3. Overlap labelled $H(X,Y)$

4. Left-only region labelled H(X|Y)

5. Arrows and plus/minus signs between regions

1. The diagram remains static while the lecturer discusses conditional entropy and code length.

1. The overlap consistently represents joint entropy.

2. The left-only region consistently represents conditional entropy of X given Y.

The picture conveys that joint uncertainty can be partitioned into what is shared with Y and what remains specifically in X after Y is known, matching H(X|Y)=$H(X,Y)-H(Y)$.

1. Diagram
   Observation

   Right-side table lists i, $a_i$, $p_i$, $h(p_i)$ for English letters and ends with the formula $\sum_i p_i \log_2 \frac{1}{p_i}$ and caption 'Table 2.9 Shannon information contents of the consonant n'.

1. The caption wording appears unusual relative to the full alphabet table; the clip does not explain this discrepancy.

1. Columns i, $a_i$, $p_i$, $h(p_i)$

2. Rows for letters a through z

3. Bottom summation formula $\sum_i p_i \log_2 \frac{1}{p_i}$

4. Caption 'Table 2.9 Shannon information contents of the consonant n'

1. The table stays visible throughout the first slide while the lecturer discusses coding English characters.

1. Each row associates a letter with a probability and a corresponding $h(p_i)$ value.

2. The base-2 logarithm formula remains displayed at the bottom.

The table supplies empirical letter probabilities and their per-symbol information contents, supporting the broader discussion of entropy and coding for English text.

1. Animation
   Observation

   The presentation changes from the conditional-entropy slide to an exercise slide retaining the title 'Conditional entropy'.

1. Exact transition frame is approximate due to sampling.

1. Previous conditional-entropy slide

2. New exercise slide titled 'Conditional entropy'

3. Venn diagram moved to top-right corner on exercise slide

1. Main body text switches from definitions/example to 'Exercise: Let's code it!' with subparts a-e.

1. The topic remains conditional entropy.

2. The Venn diagram continues to appear as a visual reference.

The lecture shifts from conceptual explanation to implementation-oriented practice.

1. Diagram
   Observation

   Final slide title: 'Introduction to Information Theory: summary' with bullets about uncertainty and surprise, entropy as -p log p, calculating entropy, and moving on to measuring information.

1. Title 'Introduction to Information Theory: summary'

2. Bullet 'We've been introduced to the ideas of uncertainty and surprise.'

3. Bullet 'Understand the meaning of entropy as -p log p'

4. Bullet 'Know how to calculate entropy'

5. Bullet 'Coming up: Move onto measuring information'

1. The slide remains static while the lecturer recaps the first lecture.

1. The summary emphasizes uncertainty, surprise, entropy, and the next topic of measuring information.

This closing visual frames conditional entropy as part of a broader introductory treatment of information theory before moving to mutual-information-style topics.

1. Caption evidence
   Observation

   Static slide titled 'Introduction to Information Theory: summary' with bullet points summarizing concepts of uncertainty, surprise, entropy (-p log p), and upcoming topics on measuring information.

1. Text slide

1. Slide content remains static throughout the clip.

The visual serves as a structural outline for the spoken summary, reinforcing the key mathematical formulas and conceptual takeaways of the preceding lecture section.

1. Audio
   Observation

   Speaker explicitly warns: 'notice we're not adding the two circles because that would double count this overlapping section'.

Believing that Joint Entropy $H(X,Y)$ is simply the sum of Marginal Entropies $H(X) + H(Y)$.

Adding the full areas of both circles counts the mutual information (intersection) twice. The correct joint entropy is the area of the union, which equals $H(X) + H(Y) - I$(X;Y).

1. Audio
   Observation

   it does not increase the uncertainty I should say on average. It may leave the uncertainty the same, but it's not going to increase it on average.

Learning additional information Y might increase uncertainty about X.

On average, conditional entropy never exceeds unconditional entropy: H(X|Y) ≤ $H(X)$. Knowledge can only reduce or maintain uncertainty, never increase it on average.

1. Audio
   Observation

   The lecturer says conditional entropy is 'generally lower' than unconditional entropy in the English-text example.

1. This caution is added by the analyst and is not explicitly stated in the clip.

One might take 'context reduces uncertainty' as an absolute law for all random variables.

The video only asserts a typical reduction for dependent sources such as English text. In general information theory, conditioning can leave entropy unchanged and, for continuous variables, can even increase differential entropy; the clip does not discuss those exceptions.

1. Diagram
   Observation

   The slide uses a Venn diagram to depict $H(X)$, $H(Y)$, $H(X,Y)$, and H(X|Y).

1. This caution is analyst-added; the lecturer does not warn against over-reading the diagram.

Students may think the circles represent ordinary sets of outcomes whose areas directly encode probabilities.

Here the diagram is a visual analogy for additive relationships among entropy quantities, especially H(X|Y)=$H(X,Y)-H(Y)$, not a claim that entropies are set cardinalities.

1. Formula
   Observation

   The slide shows H(X|Y) = $H(X,Y) - H(Y)$.

The conditional entropy is derived from and depends on the concepts of joint entropy and marginal entropy.

1. Audio
   Observation

   The speaker transitions from discussing the specific surprise h(x|y) to the average uncertainty H(X|Y).

Conditional entropy is the expected value (average) of the conditional surprisal.

1. Formula
   Observation

   Slide shows H(X|Y) = $H(X,Y) - H(Y)$.

The definition of conditional entropy is algebraically linked to joint and marginal entropies via subtraction.

1. Audio
   Observation

   Transition from discussing formulas to 'We can think about them graphically... draw Venn diagrams'.

The analytical definition is contrasted with a geometric visualization to build intuition about how information overlaps.

1. Formula
   Observation

   H(X|Y) = $H(X,Y) - H(Y)$

The definition of conditional entropy is equivalent to the chain rule decomposition of joint entropy

1. Formula
   Observation

   $0 \le H$(X|Y) $\le H(X)$

2. Formula
   Observation

   H(X|Y) = $H(X) \text{ iff } X \text{ and } Y \text{ are independent}$

The equality case H(X|Y) = $H(X)$ in the bounds represents the special case where X and Y are independent

1. Audio
   Observation

   all the information in X would be contained inside Y

Zero conditional entropy represents the extreme case where X is completely determined by Y

1. Formula
   Observation

   The definition line and the identity H(X|Y)=$H(X,Y)-H(Y)$ appear together on the same slide.

The chain-rule formula operationalizes the verbal definition by expressing conditional entropy in terms of joint and marginal entropies.

1. Formula
   Observation

   The slide lists h(x|y)=$h(x,y)-h(y)$ directly above H(X|Y)=$H(X,Y)-H(Y)$.

1. The averaging step from h to H is not explicitly derived in the clip.

The pointwise self-information identity is presented alongside its expected-value counterpart for entropies.

1. Caption evidence
   Observation

   Slide bullet links previous-character context to changed probabilities and names Markov chains.

2. Audio
   Observation

   Lecturer uses the Markov-chain idea to motivate why H(X|Y) matters for text.

Markov-chain context provides a concrete source model in which conditional entropy measures the remaining uncertainty of the next symbol.

1. Caption evidence
   Observation

   Slide explicitly says H(X|Y) drops 'therefore the code length for a conditional encoding of incoming character X'.

2. Audio
   Observation

   Lecturer repeats that using context reduces the amount of code per letter.

Conditional entropy is used as the quantity that determines how much shorter a conditional code can be.

1. Caption evidence
   Observation

   Exercise asks students to implement conditionalentropy(p), with hints to reuse entropy() and jointentropy().

The programming task relies on the same decomposition of conditional entropy into joint and marginal pieces taught just before.

1. Audio
   Observation

   The speaker introduces the concept to quantify uncertainty left in X given what we already know.

1. Conditional Entropy

1. Formula
   Observation

   The slide displays the summation formula for H(X|Y).

1. Conditional Entropy

1. Formula
   Observation

   The slide shows H(X|Y) = $H(X,Y) - H(Y)$.

1. Conditional Entropy
2. Joint Entropy
3. Marginal Entropy

1. Formula
   Observation

   Explicit formula on slide.

1. Definition of Conditional Entropy

1. Diagram
   Observation

   Venn diagram showing union.

1. Venn Diagram Representation of Entropies
2. Joint Entropy vs Sum of Marginal Entropies

1. Audio
   Observation

   Explanation of double counting.

1. Double Counting in Joint Entropy

1. Audio
   Observation

   the average conditional entropy is always less than or equal to the unconditional entropy

1. Bounds on Conditional Entropy

1. Audio
   Observation

   that is the case if our variables X and Y are independent

1. Equality Condition for Independence

1. Audio
   Observation

   there is no surprise left in X once we know Y

1. Zero Conditional Entropy Interpretation

1. Diagram
   Observation

   Venn diagram with labeled regions

1. Venn Diagram of Entropy Relationships
2. Venn Diagram Representation of Entropies

1. Caption evidence
   Observation

   Slide definition of conditional entropy.

1. Definition of conditional entropy

1. Formula
   Observation

   Displayed identity H(X|Y)=$H(X,Y)-H(Y)$.

2. Diagram
   Observation

   Venn diagram showing overlap and left-only region.

1. Entropy chain rule for conditional entropy