Dimensionality Reduction with PCA: How Principal Component Analysis Works

1. Formula
   Observation

   The formula $\mathbf{x} = [x_1, x_2, \dots, x_D]^T \in \mathbb{R}^D$ is displayed on the screen.

2. Audio
   Observation

   The narrator mentions processing complex measurements with dozens of variables at once.

$\mathbf{x} = [x_1, x_2, \dots, x_D]^T \in \mathbb{R}^D$

A column vector representing a data point with D features in a D-dimensional real space.

$\mathbb{R}^D$

1. Formula
   Observation

   The text 'x : Nose Length' is shown in the Measured Coordinates box.

x

The first coordinate representing Nose Length in the 3D feature space.

$\mathbb{R}$

1. Formula
   Observation

   The text 'y : Eye Distance' is shown in the Measured Coordinates box.

y

The second coordinate representing Eye Distance in the 3D feature space.

$\mathbb{R}$

1. Formula
   Observation

   The text 'z : Jaw Width' is shown in the Measured Coordinates box.

z

The third coordinate representing Jaw Width in the 3D feature space.

$\mathbb{R}$

1. Formula
   Observation

   The text 'Person i : (1.5, 1.2, 0.9)' is displayed.

(1.5, 1.2, 0.9)

A specific data point representing a person's facial measurements.

$\mathbb{R}^3$

1. Formula
   Observation

   The label 'PC 1 (Max Spread)' appears next to a blue arrow.

$\text{PC 1}$

The first principal component, representing the direction of maximum variance or spread in the data cloud.

$\mathbb{R}^3$

1. Formula
   Observation

   The label 'PC 2' appears next to a cyan arrow perpendicular to PC 1.

$\text{PC 2}$

The second principal component, representing the next widest perpendicular direction of variance.

$\mathbb{R}^3$

1. Formula
   Observation

   Bottom-right formula box shows $d_\perp = \min_{\mathbf{p} \in \mathcal{P}}$ \|$\mathbf{x} - \mathbf{p}$\|.

2. Audio
   Observation

   Narrator says the perpendicular drop lines show the precise distance sacrificed during compression.

$d_\perp$

Perpendicular distance from a point to the chosen plane, i.e. the amount of information lost by orthogonal projection.

Nonnegative real distance associated with a point $\mathbf{x}$ and plane $\mathcal{P}$.

1. Formula
   Observation

   Formula uses $\mathbf{x}$ inside \|$\mathbf{x} - \mathbf{p}$\|.

2. Diagram
   Observation

   Blue points in the 3D scene are the objects being projected onto the green plane.

$\mathbf{x}$

Original data point in the ambient 3D space before projection.

Point/vector in $\mathbb{R}^3$.

1. Formula
   Observation

   Minimization is over $\mathbf{p} \in \mathcal{P}$.

2. Diagram
   Observation

   Yellow landing markers on the green plane correspond to projected positions.

$\mathbf{p}$

Candidate point on the target plane used in the minimization defining the perpendicular distance.

Point/vector in the plane $\mathcal{P}$.

1. Formula
   Observation

   The subscript in the minimization is $\mathbf{p} \in \mathcal{P}$.

2. Diagram
   Observation

   A translucent green plane is the surface receiving the projected points.

$\mathcal{P}$

Chosen 2D plane (principal plane) onto which the 3D data are orthogonally projected.

A plane subset of $\mathbb{R}^3$.

1. Formula
   Observation

   The expression contains \|$\mathbf{x} - \mathbf{p}$\|.

\|$\cdot$\|

Vector norm measuring Euclidean length of the difference vector.

Maps vectors to nonnegative real numbers.

1. Audio
   Observation

   Narrator explains squishing a rich 3D diorama flat into a 2D photograph, noting that an awkward angle causes overlap while the right angle preserves essential detail.

2. Animation
   Observation

   A 3D point cloud is projected onto a 2D plane. First, it projects onto a plane causing overlapping points (Awkward Angle), then onto a tilted plane preserving structure (Optimal Projection).

Dimensionality reduction can be visualized as casting a shadow of a high-dimensional object onto a lower-dimensional surface. The choice of the projection plane determines whether the structural information of the data is preserved or lost in overlapping blur.

1. High-dimensional data must be projected onto a lower-dimensional subspace.

2. The projection should ideally preserve the maximum possible variance or structure of the original data.

1. Audio
   Observation

   Narrator describes natural human facial proportions correlating with each other, causing points to cluster into an elongated, tilted cloud rather than scattering randomly like static.

2. Formula
   Observation

   Text boxes display 'Hypothesis: Uncorrelated Static', 'Reality: Tilted Correlated Cloud', and 'Dominant Variance Orientation: Redundant dimensions can be compressed'.

In real-world datasets, features are often correlated. When plotted in a multi-dimensional space, these correlations cause the data points to form an elongated, tilted cloud rather than a spherical or random distribution. This dominant variance orientation indicates that some dimensions are redundant and can be compressed.

1. The dataset contains multiple numerical features.

2. The features exhibit natural correlations with each other.

1. Audio
   Observation

   Narrator states the target is to find the exact 2D surface capturing the greatest possible diversity, accomplished by identifying the direction of widest spread followed by the next widest perpendicular direction.

2. Animation
   Observation

   Two orthogonal vectors (PC 1 and PC 2) are drawn through the center of the point cloud, defining a semi-transparent blue plane labeled 'Optimal 2D Projection Canvas'.

Principal Component Analysis finds the optimal lower-dimensional subspace by identifying orthogonal directions of maximum variance. The first principal component (PC 1) aligns with the widest spread of the data, and subsequent components (like PC 2) align with the next widest perpendicular spreads. Together, they define the optimal projection canvas.

1. The data must be centered (implied by vectors originating from the cloud's center).

2. The principal components must be mutually orthogonal.

1. Geometry of Correlated Feature Clouds

1. Audio
   Observation

   Narrator explains that every 3D point travels along the shortest conceivable path until it lands flat on the surface, perpendicular to the chosen plane.

2. Diagram
   Observation

   Blue points move vertically down to yellow markers on a green plane; dashed drop lines remain visible.

3. Formula
   Observation

   Formula box defines $d_\perp = \min_{\mathbf{p} \in \mathcal{P}}$ \|$\mathbf{x} - \mathbf{p}$\|.

The segment defines orthogonal projection geometrically: each 3D point is moved straight down along the direction perpendicular to the chosen plane until it reaches the plane. The narrator emphasizes that this is not random axis deletion but a shortest-path drop that discards only the depth variation perpendicular to the preferred canvas.

1. A target plane $\mathcal{P}$ has been chosen.

2. Projection is perpendicular to that plane.

3. The quantity being discarded is the component of $\mathbf{x}$ normal to $\mathcal{P}$.

1. Formula
   Observation

   Displayed formula explicitly defines $d_\perp$ via a minimum over points in $\mathcal{P}$.

2. Audio
   Observation

   Narrator states that the perpendicular drop lines show the precise distance sacrificed during compression.

The formula identifies the loss caused by projection with the minimal Euclidean distance from the original point $\mathbf{x}$ to any point $\mathbf{p}$ on the plane $\mathcal{P}$. In this geometric setting, that minimum is realized at the orthogonal foot of the perpendicular.

1. $\mathbf{x}$ is the original point.

2. $\mathcal{P}$ is the target plane.

3. \|$\cdot$\| is a vector norm measuring distance.

1. Orthogonal shadow projection onto a plane

1. Audio
   Observation

   Narrator says the entire 3D cloud collapses in unison onto the two-dimensional surface and each point glides downward along its trailing path.

2. Diagram
   Observation

   Multiple vertical clusters of blue points descend to the green plane, leaving dashed trails and forming planar clusters.

3. Caption evidence
   Observation

   On-screen text includes 'The Collapse Onto the 2D Surface' and later '2D Inspection Frame Ready' with ($x_1$,$x_2$,$x_3$) -> ($x_1$,$x_2$).

1. The displayed ($x_1$,$x_2$,$x_3$) -> ($x_1$,$x_2$) is a simplified inspection-frame notation shown on screen; the narration elsewhere frames the method as projection onto principal components rather than literal coordinate truncation.

This section visualizes dimensionality reduction as a collective motion: many 3D points fall along parallel perpendicular paths onto a plane, producing a 2D arrangement that can be inspected more easily. The analogy card reinforces that thickness disappears while planar pattern remains.

1. Input is a set of points in 3D.

2. A 2D plane is selected as the inspection surface.

3. Projection is along the normal direction to that plane.

1. Orthogonal shadow projection onto a plane

1. Audio
   Observation

   Narrator says the plane is extracted into its own independent two-dimensional coordinate system; left side needs three numbers per face, right side needs two.

2. Formula
   Observation

   Left panel shows $x \in \mathbb{R}^3$ with [x_nose, x_eye, x_jaw]^T; right panel shows $z \in \mathbb{R}^2$ with [$z_1$, $z_2$]^$T = [[x \cdot e_1],[x \cdot e_2]]$.

3. Diagram
   Observation

   Side-by-side panels compare the original 3D feature space and the extracted 2D principal plane.

The video converts the geometric projection into an explicit coordinate representation. A face originally described by three measurements is re-expressed by two scalar coordinates, namely its projections onto the principal component directions $e_1$ and $e_2$. The side-by-side display stresses that cluster structure is preserved while storage shrinks.

1. Original sample $\mathbf{x}$ lies in $\mathbb{R}^3$.

2. Principal directions $\mathbf{e}_1$,$\mathbf{e}_2$ define the 2D coordinate system.

3. Coordinates are obtained by dot products with those directions.

1. Orthogonal shadow projection onto a plane
2. Collapse of a 3D cloud onto a 2D surface

1. Caption evidence
   Observation

   Bottom banner reads 'Storage Reduction: -33.3% (3 floats -> 2 floats per face)'.

2. Audio
   Observation

   Narrator says storage requirement is reduced by thirty-three percent while preserving relative clusters.

Because each face changes from three stored real numbers to two, the video quantifies the saving as a 33.3% reduction in storage per sample. It also claims that the relative cluster separation among subjects is preserved in the lower-dimensional view.

1. Each original sample stores 3 floating-point coordinates.

2. Each compressed sample stores 2 floating-point coordinates.

1. Side-by-side coordinate compression from 3 features to 2 PCs

1. Audio
   Observation

   Narrator asks whether compression can be reversed and says we multiply the two-dimensional coordinates back through principal component directions to lift each point into 3D.

2. Formula
   Observation

   Top-right formula reads $\hat{\mathbf{x}} = \mathbf{W}\mathbf{z} = z_1\mathbf{v}_1 + z_2\mathbf{v}_2$.

3. Diagram
   Observation

   Blue arrows connect yellow 2D points on the right to cyan reconstructed points on the left.

1. The video does not state whether $\mathbf{v}_1$,$\mathbf{v}_2$ are exactly the same as the earlier $\mathbf{e}_1$,$\mathbf{e}_2$; it presents them as the principal component directions used for reconstruction.

The final section introduces reconstruction: after compression discards information, the video asks whether the journey can be reversed. It answers by showing a linear lift from $\mathbb{R}^2$ back to $\mathbb{R}^3$, using the stored coordinates as weights on the principal component directions. The result is an approximated 3D point $\hat{\mathbf{x}}$, not necessarily the exact original.

1. A compressed coordinate vector $\mathbf{z} \in \mathbb{R}^2$ is available.

2. Reconstruction uses principal component directions $\mathbf{v}_1$,$\mathbf{v}_2$.

3. The output is an approximation $\hat{\mathbf{x}}$ in $\mathbb{R}^3$.

1. Side-by-side coordinate compression from 3 features to 2 PCs

1. Formula
   Observation

   Formula box displays $\hat{x} = Wz = z_1v_1 + z_2v_2$.

2. Audio
   Observation

   Narrator explains how flat two-dimensional coordinates translate directly back into three-dimensional space.

The video presents the formula for reconstructing a 3D point from its 2D compressed coordinates. The reconstructed point $\hat{x}$ is calculated by multiplying the matrix of principal component vectors $W$ by the compressed coordinate vector $z$, which is equivalent to taking a linear combination of the principal component vectors weighted by their respective coordinates.

1. The original data must be projected onto the subspace spanned by the principal components.

2. The reconstruction will only lie within this lower-dimensional subspace.

1. Formula
   Observation

   Formula box displays Retained Essence = $\frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^d \lambda_j} \approx 100\%$.

2. Audio
   Observation

   Narrator states that you can confidently judge whether a compressed representation preserves the true essence of your data.

The video provides a mathematical criterion for evaluating dimensionality reduction. The 'Retained Essence' is defined as the ratio of the sum of the eigenvalues of the retained principal components to the sum of all eigenvalues in the original space. A value close to 100% indicates that the reduced representation captures the dominant structure of the data while discarding orthogonal background noise.

1. The eigenvalues $\lambda$ correspond to the variance captured by each principal component.

2. The sum of all eigenvalues represents the total variance in the original dataset.

1. Formula
   Observation

   Retained Essence = ($\sum_{i=1}^k \lambda_i$) / ($\sum_{j=1}^d \lambda_j$) $\approx 100$%

2. Audio
   Observation

   No audio during this segment.

The formula calculates the proportion of total variance retained when reducing the dimensionality of a dataset from d to k using Principal Component Analysis. The goal is to retain approximately 100% of the essential information (dominant structure) while discarding orthogonal background noise.

1. $\lambda_i$ represents the eigenvalues of the covariance matrix.

2. k is the number of retained principal components.

3. d is the total number of original dimensions.

1. Audio
   Observation

   Narrator states: 'Dimensionality reduction is literally the process of casting an orthogonal shadow.'

1. The audio cuts off mid-sentence at the end of the clip ('If we shine a light directly perpendic...').

Dimensionality reduction is mathematically equivalent to casting an orthogonal shadow of the high-dimensional data onto a lower-dimensional subspace.

1. The projection must be orthogonal (perpendicular to the target subspace).

Universal for linear dimensionality reduction.

1. Audio
   Observation

   Narrator says: 'Notice that we are not randomly discarding the third axis. Rather, we are sliding each point along a trajectory that ignores only the depth variation perpendicular to our preferred canvas.'

Orthogonal projection onto the chosen plane removes only the component perpendicular to that plane; it is not equivalent to arbitrarily discarding one original coordinate axis.

1. A specific target plane has been chosen.

2. Movement is along the perpendicular direction to that plane.

For every 3D point shown in the projection sequence.

1. Audio
   Observation

   Narrator says the perpendicular drop lines show the precise distance sacrificed during compression.

2. Formula
   Observation

   Formula box defines $d_\perp = \min_{\mathbf{p} \in \mathcal{P}}$ \|$\mathbf{x} - \mathbf{p}$\|.

The amount of geometric information lost when projecting a point to the plane is measured by the perpendicular distance from the point to the plane.

1. Projection is orthogonal onto plane $\mathcal{P}$.

2. Distance is measured by the displayed norm.

For each point $\mathbf{x}$ being projected.

1. Audio
   Observation

   Narrator compares the collapse to pressing a 3D flower between pages: physical thickness vanishes, yet perimeter, petal arrangement, and internal boundaries stay visible on the flattened sheet.

2. Caption evidence
   Observation

   Analogy card reads '3D Volume -> Planar Pattern' and 'Thickness -> 0 | Pattern Kept'.

In the pressed-flower analogy, orthogonal collapse eliminates the third-dimensional thickness while retaining the visible 2D arrangement of features.

1. The object is flattened along one direction onto a plane.

2. Attention is restricted to the resulting planar pattern.

Stated for the illustrative analogy, not as a formal theorem.

1. Audio
   Observation

   Narrator says: 'Because compression inevitably discards information, can we reverse the journey and rebuild our original measurements?'

Reducing dimensionality by projection loses information from the original data.

1. A higher-dimensional dataset is mapped to a lower-dimensional representation.

General statement made by the narrator for the compression process under discussion.

1. Caption evidence
   Observation

   Banner states 'Storage Reduction: -33.3% (3 floats -> 2 floats per face)'.

2. Audio
   Observation

   Narrator says storage requirement is reduced by thirty-three percent while preserving relative clusters.

Changing the representation of each face from three stored floats to two stored floats reduces storage per face by 33.3%.

1. Each original face is stored with 3 floating-point numbers.

2. Each compressed face is stored with 2 floating-point numbers.

Per face/sample in the displayed example.

1. Audio
   Observation

   Narrator states: 'The third principal component, which we chose to drop, pointed precisely along the direction of these red lines.'

2. Diagram
   Observation

   Text appears saying 'PC3 = Error Direction' with an arrow pointing along the vertical axis connecting the original and reconstructed points.

The reconstruction error in PCA points precisely along the direction of the dropped principal components.

1. A subset of principal components has been dropped during dimensionality reduction.

2. The data is reconstructed using only the retained principal components.

For any dropped principal component direction.

1. Audio
   Observation

   Narrator states: 'When PCA reports 95% variance explained, it means our data was barely three-dimensional to begin with.'

2. Diagram
   Observation

   Visuals show a flat 2D plane with points hovering extremely close to it, labeled 'Subspace thickness ~ 0'.

A high percentage of variance explained by a lower-dimensional subspace implies that the original data had very little spread or depth in the discarded dimensions.

1. The data is projected onto a lower-dimensional subspace.

2. The variance explained is calculated based on the eigenvalues of the principal components.

For datasets where the retained subspace captures a large majority of the total variance.

1. Audio
   Observation

   Narrator explains finding the exact 2D surface that captures the greatest possible diversity by identifying the direction of widest spread, followed by the next widest perpendicular direction.

2. Animation
   Observation

   Visual demonstration of PC 1 (Max Spread) and PC 2 forming a plane that aligns with the elongated data cloud, contrasted with a suboptimal angle causing severe point clumping.

1. Expression

   $$$$
   Explanation

   Identify the direction of maximum variance in the 3D data cloud.

   Justification

   This direction captures the most information or 'diversity' among the subjects, corresponding to the longest axis of the tilted cloud.

   Shown in the video
2. Expression

   $$$$
   Explanation

   Identify the next direction of maximum variance that is strictly perpendicular to the first.

   Justification

   Orthogonality ensures that the new dimension captures entirely new information not already represented by the first principal component.

   Shown in the video
3. Expression

   $$$$
   Explanation

   Define the 2D plane spanned by these two orthogonal eigenvectors.

   Justification

   This plane acts as the optimal projection canvas, minimizing information loss and preventing data points from clumping together.

   Shown in the video

The plane defined by the top two principal components provides the optimal 2D surface for maximizing the spread and preserving the structure of the original 3D data.

1. Formula
   Observation

   The displayed equation defines $d_\perp$ as a minimum over points in the plane.

2. Diagram
   Observation

   Vertical dashed segments connect original points to their feet on the plane.

1. Expression

   $$d_\perp = \min_{\mathbf{p} \in \mathcal{P}} \|\mathbf{x} - \mathbf{p}\|$$
   Explanation

   The video introduces the perpendicular distance as the minimum distance from the original point $\mathbf{x}$ to any point $\mathbf{p}$ lying on the plane $\mathcal{P}$.

   Justification

   Directly shown in the on-screen formula.

   Shown in the video
2. Expression

   $$$$
   Explanation

   The animation then identifies this minimizing point with the foot of the perpendicular drop from $\mathbf{x}$ to the plane.

   Justification

   Visual correspondence between the dashed drop line and the formula's minimization over the plane.

   Derived from the video
3. Expression

   $$$$
   Explanation

   Therefore the length of the drop line is interpreted as the exact amount of distance sacrificed by projection.

   Justification

   Narration explicitly links the drop lines to the distance sacrificed during compression.

   Shown in the video

The perpendicular drop length visualized in the animation is formalized as $d_\perp$, the minimal distance from $\mathbf{x}$ to the plane $\mathcal{P}$.

1. Formula
   Observation

   Right panel displays [$z_1$,$z_2$]^$T = [[x \cdot e_1],[x \cdot e_2]]$.

2. Audio
   Observation

   Narrator says the plane is extracted into its own independent two-dimensional coordinate system and each face now needs only two numbers.

1. Expression

   $$\mathbf{x} \in \mathbb{R}^3,\quad \mathbf{x} = [x_{nose}, x_{eye}, x_{jaw}]^T$$
   Explanation

   The left panel starts from the original three-measurement description of a face.

   Justification

   Shown in the side-by-side display and named in the narration.

   Shown in the video
2. Expression

   $$\mathbf{z} \in \mathbb{R}^2$$
   Explanation

   The right panel introduces a new two-dimensional coordinate system attached to the extracted principal plane.

   Justification

   Explicitly labeled on screen.

   Shown in the video
3. Expression

   $$\begin{bmatrix} z_1 \\ z_2 \end{bmatrix} = \begin{bmatrix} \mathbf{x} \cdot \mathbf{e}_1 \\ \mathbf{x} \cdot \mathbf{e}_2 \end{bmatrix}$$
   Explanation

   Each compressed coordinate is obtained by taking the dot product of the original vector with one principal direction.

   Justification

   Displayed formula in the right panel.

   Shown in the video
4. Expression

   $$$$
   Explanation

   Thus the 3D sample is represented by two scalar projections instead of three original measurements.

   Justification

   Follows from the displayed definition of $\mathbf{z}$.

   Derived from the video

The side-by-side derivation shows that compression replaces the triple (x_nose, x_eye, x_jaw) by the pair ($z_1$, $z_2$) of projections onto principal directions.

1. Formula
   Observation

   Top-right formula reads $\hat{\mathbf{x}} = \mathbf{W}\mathbf{z} = z_1\mathbf{v}_1 + z_2\mathbf{v}_2$.

2. Audio
   Observation

   Narrator says multiplying the 2D coordinates back through principal component directions lifts each point into 3D.

3. Animation
   Observation

   Blue arrows carry yellow 2D points to cyan reconstructed 3D points.

1. The exact relationship between $\mathbf{W}$ and the previously shown $\mathbf{e}_1$,$\mathbf{e}_2$ is not verbally spelled out in this clip.

1. Expression

   $$\mathbf{z} \in \mathbb{R}^2 \rightarrow \hat{\mathbf{x}} \in \mathbb{R}^3$$
   Explanation

   The lower-right label states that the map sends compressed 2D coordinates to reconstructed 3D vectors.

   Justification

   Shown on screen beneath the 2D panel.

   Shown in the video
2. Expression

   $$\hat{\mathbf{x}} = \mathbf{W}\mathbf{z}$$
   Explanation

   Reconstruction is presented as multiplication of the coordinate vector by a matrix $\mathbf{W}$.

   Justification

   Displayed formula at top right.

   Shown in the video
3. Expression

   $$\hat{\mathbf{x}} = z_1\mathbf{v}_1 + z_2\mathbf{v}_2$$
   Explanation

   The same formula expands the matrix product into a linear combination of two direction vectors weighted by the stored coordinates.

   Justification

   Displayed equality in the same formula box.

   Shown in the video
4. Expression

   $$$$
   Explanation

   The animation then realizes this algebra by drawing arrows from each 2D point to a corresponding lifted 3D point.

   Justification

   Visible blue transfer arrows and color change from yellow to cyan.

   Shown in the video

The clip derives reconstruction as a linear lift: the 2D coordinates are recombined along principal directions to produce an approximated 3D vector $\hat{\mathbf{x}}$.

1. Audio
   Observation

   Narrator explains the process of lifting shadows back to 3D and notes that reconstructed points sit trapped on the exact 2D slice.

2. Animation
   Observation

   Animation shows 2D yellow points being mapped back to 3D cyan points that all lie perfectly flat on a green 2D plane within the 3D space.

1. Expression

   $$$$
   Explanation

   Start with the 2D compressed coordinates $z$ for a data point.

   Justification

   Given from the previous projection step.

   Shown in the video
2. Expression

   $$\hat{x} = Wz = z_1v_1 + z_2v_2$$
   Explanation

   Apply the reconstruction formula by multiplying the coordinates by the principal component matrix $W$.

   Justification

   Definition of PCA reconstruction.

   Shown in the video
3. Expression

   $$$$
   Explanation

   The resulting vector $\hat{x}$ is a linear combination of the basis vectors $v_1$ and $v_2$.

   Justification

   Algebraic expansion of matrix-vector multiplication.

   Derived from the video
4. Expression

   $$$$
   Explanation

   Therefore, $\hat{x}$ must lie entirely within the 2D subspace spanned by $v_1$ and $v_2$.

   Justification

   Definition of a span in linear algebra.

   Derived from the video

Reconstructed points from PCA are strictly confined to the lower-dimensional subspace defined by the retained principal components, meaning they cannot recover the original heights or depths outside this slice.

1. Audio
   Observation

   Narrator describes the gap between the original point and the reconstructed point as the reconstruction error, visualized as red connecting whiskers.

2. Animation
   Observation

   Animation shows red vertical lines connecting floating 3D points to their corresponding projections on the 2D plane.

1. Expression

   $$e_i = ||x_i - \hat{x}_i||$$
   Explanation

   Define the reconstruction error for a single point as the Euclidean distance between the original 3D point and its 2D reconstruction.

   Justification

   Standard definition of reconstruction error.

   Shown in the video
2. Expression

   $$$$
   Explanation

   Observe that the vector difference $x_i - \hat{x}_i$ is perpendicular to the 2D subspace.

   Justification

   Property of orthogonal projection.

   Derived from the video
3. Expression

   $$$$
   Explanation

   Identify the direction of this perpendicular vector with the dropped principal component (PC3).

   Justification

   Principal components are mutually orthogonal.

   Shown in the video
4. Expression

   $$\sum ||x_i - \hat{x}_i||^2 = 8.75$$
   Explanation

   Sum the squared errors across all points to get the Total Error (SSE).

   Justification

   Definition of Sum of Squared Errors.

   Shown in the video

The reconstruction error physically represents the ignored depth or thickness of the data along the axes of the discarded principal components.

1. Audio
   Observation

   Narrator introduces a vision system analyzing human faces across three numerical features: nose length, eye distance, and jaw width.

2. Formula
   Observation

   Text boxes list 'x : Nose Length', 'y : Eye Distance', 'z : Jaw Width', and a specific example 'Person i : (1.5, 1.2, 0.9)'.

Represent human facial measurements in a 3D feature space and observe their geometric distribution.

1. Features: Nose Length (x), Eye Distance (y), Jaw Width (z).

2. Example data point: Person $i = (1.5, 1.2, 0.9)$.

Visualize how natural correlations affect the geometry of the data cloud.

1. Expression

   $$\mathbf{x}_i = (1.5, 1.2, 0.9)$$
   Explanation

   Map a single person's facial measurements to a coordinate in 3D space.

   Justification

   Each person becomes a single data point in the feature space.

   Shown in the video
2. Expression

   $$$$
   Explanation

   Observe the distribution of many such points.

   Justification

   Because natural human facial proportions correlate with each other, the points do not scatter randomly like static; instead, they cluster into an elongated, tilted cloud.

   Shown in the video

The data forms a tilted, correlated 3D cloud with a dominant variance orientation, indicating that redundant dimensions can be compressed.

Visual confirmation of the elongated yellow/white point cloud stretching along a specific diagonal axis rather than uniformly filling the 3D space.

1. Audio
   Observation

   Narrator says the original data required three numbers per face: nose length, eye distance, and jaw width; on the right every face needs only two numbers.

2. Formula
   Observation

   Left panel shows [x_nose, x_eye, x_jaw]^T; right panel shows [$z_1$, $z_2$]^$T = [[x \cdot e_1],[x \cdot e_2]]$.

3. Diagram
   Observation

   Colored point clusters in 3D are mirrored as corresponding clusters in the 2D principal plane.

Represent each face first by three anatomical measurements and then by two principal-component coordinates, while comparing the resulting storage and cluster structure.

1. Original feature vector per face: [x_nose, x_eye, x_jaw]^$T \in \mathbb{R}^3$.

2. Compressed representation: [$z_1$, $z_2$]^$T \in \mathbb{R}^2$.

3. $z_1 = \mathbf{x} \cdot \mathbf{e}_1$ and $z_2 = \mathbf{x} \cdot \mathbf{e}_2$.

Show how the 3D feature description is reduced to 2D coordinates and quantify the storage saving.

1. Expression

   $$\mathbf{x} = [x_{nose}, x_{eye}, x_{jaw}]^T$$
   Explanation

   Start from the original three-number description of a face in 3D feature space.

   Justification

   Given in the left panel and narration.

   Shown in the video
2. Expression

   $$\begin{bmatrix} z_1 \\ z_2 \end{bmatrix} = \begin{bmatrix} \mathbf{x} \cdot \mathbf{e}_1 \\ \mathbf{x} \cdot \mathbf{e}_2 \end{bmatrix}$$
   Explanation

   Compute the two compressed coordinates as projections of $\mathbf{x}$ onto the principal directions.

   Justification

   Displayed formula in the right panel.

   Shown in the video
3. Expression

   $$3 \text{ floats} \rightarrow 2 \text{ floats per face}$$
   Explanation

   Compare the number of stored real values before and after compression.

   Justification

   Explicitly stated in the bottom banner and narration.

   Shown in the video
4. Expression

   $$\frac{3-2}{3}=33.3\%$$
   Explanation

   Convert the count change into the reported percentage reduction in storage per face.

   Justification

   Matches the on-screen '-33.3%' claim.

   Derived from the video

Each face is represented by two principal coordinates instead of three raw measurements, yielding a 33.3% reduction in storage per face while preserving relative cluster structure in the displayed example.

The side-by-side visualization shows corresponding colored clusters in both panels, supporting the claim that subject grouping is retained after compression.

1. Audio
   Observation

   Narrator uses the example of PCA reporting 95% variance explained to illustrate that the data was barely 3D to begin with.

2. Diagram
   Observation

   Diagram shows 'Plane (PC1+PC2): 95%' and '3rd Dim: 5%', alongside a visual of points tightly clustered on a flat plane.

What does it mean when PCA reports that 95% of the variance is explained by a 2D projection of 3D data?

1. Original data is 3-dimensional.

2. PCA projects the data onto a 2-dimensional plane (PC1 and PC2).

3. The retained variance is 95%.

Explain the physical and geometric meaning of this high variance retention.

1. Expression

   $$$$
   Explanation

   Analyze the side angle of the data cloud relative to the 2D projection plane.

   Justification

   Visualizing the geometry helps understand variance distribution.

   Shown in the video
2. Expression

   $$$$
   Explanation

   Observe that the points hover extremely close to the 2D plane, indicating a subspace thickness of approximately zero.

   Justification

   Direct observation from the provided diagram.

   Shown in the video
3. Expression

   $$$$
   Explanation

   Conclude that the third dimension contained only 5% of the overall scatter.

   Justification

   Complement of the retained variance (100% - 95%).

   Derived from the video

It means the original data was barely three-dimensional; squishing it flat cost almost nothing because there was almost no depth to lose.

The visual diagram explicitly labels the 3rd dimension as containing only 5% of the variance, confirming the narrator's explanation.

1. Animation
   Observation

   A 3D point cloud is projected downwards onto a horizontal plane, resulting in overlapping red points labeled 'Awkward Angle: Overlapping Blur'. Then, the projection plane tilts to align with the data, resulting in distinct yellow points labeled 'Optimal Projection: Essential Structure Preserved'.

1. 3D point cloud

2. Horizontal projection plane

3. Tilted projection plane

4. Projection lines (shadows)

1. The orientation of the projection plane changes from horizontal to tilted.

2. The color of the projected points changes from red (overlapping) to yellow (distinct).

1. The original 3D positions of the data points remain constant.

Demonstrates visually that projecting high-dimensional data onto a poorly chosen subspace destroys structural information through overlap, whereas an optimally aligned subspace preserves it.

1. Animation
   Observation

   A yellow dot appears at the center of the cloud. A blue arrow labeled 'PC 1 (Max Spread)' extends along the longest axis of the cloud. A cyan arrow labeled 'PC 2' extends perpendicularly. A semi-transparent blue grid plane forms containing both arrows.

1. Centered data point (yellow dot)

2. PC 1 vector (blue arrow)

3. PC 2 vector (cyan arrow)

4. Optimal 2D Projection Canvas (blue grid plane)

1. Vectors are sequentially drawn to define the axes of maximum variance.

2. A plane is generated spanning the two vectors.

1. The underlying 3D data cloud remains visible and unchanged.

Illustrates the geometric definition of Principal Component Analysis: finding orthogonal directions of maximum variance to construct the optimal lower-dimensional subspace for projection.

1. Animation
   Observation

   A green wireframe plane is shown. Blue data points float above it. Dashed white lines drop perpendicularly from the blue points to the green plane, creating smaller blue shadow points on the surface.

1. The animation is cut off before the full explanation of the light source is completed.

1. Green wireframe projection plane

2. Blue 3D data points

3. Dashed white projection lines

4. Blue 2D shadow points

1. Data points are mapped from 3D space to the 2D plane via straight, perpendicular lines.

1. The relative ordering of points along the projection plane is maintained according to the orthogonal mapping.

Provides the literal geometric interpretation of linear dimensionality reduction: every point in the high-dimensional space is mapped to its closest point on the lower-dimensional subspace, analogous to casting an orthogonal shadow.

1. Diagram
   Observation

   Blue points hover above a translucent green plane; yellow markers appear where they land; dashed vertical segments trace the paths.

2. Audio
   Observation

   Narrator describes each point traveling along the shortest path perpendicular to the chosen plane.

1. Blue 3D points

2. Translucent green plane

3. Yellow landing markers

4. Dashed perpendicular drop lines

5. Coordinate axes

1. Points move downward toward the plane.

2. Yellow markers appear at the landing positions.

3. Dashed lines accumulate to show the traveled perpendicular paths.

1. The plane orientation stays fixed.

2. Motion is along the direction perpendicular to the plane.

3. The horizontal placement of each landing point matches the original point's in-plane position.

The animation visualizes orthogonal projection: each point is sent to the nearest point on the plane along the normal direction, and the drop-line length represents the discarded perpendicular component.

1. Diagram
   Observation

   Multiple vertical stacks of blue points descend together onto the green plane, leaving dashed trails and forming several planar clusters.

2. Caption evidence
   Observation

   Title reads 'The Collapse Onto the 2D Surface'; later cards read 'Analogy: Pressed Flower', 'What if data had 10 dimensions?', and '2D Inspection Frame Ready'.

1. Several vertical clusters of blue points

2. Green plane with grid

3. Dashed descent trails

4. Analogy card with purple flower

5. Prompt card about 10 dimensions

6. Inspection-frame card

1. Many points descend simultaneously.

2. The 3D cloud becomes a set of 2D clusters on the plane.

3. Supplementary text cards appear one after another.

1. The plane remains the target surface.

2. Each point continues moving along a perpendicular path.

3. Relative grouping of nearby points is visually maintained on the plane.

This sequence generalizes the single-point projection to an entire dataset, emphasizing that dimensionality reduction can be viewed as flattening a cloud while keeping its planar pattern.

1. Diagram
   Observation

   Split screen compares 'Original 3D Feature Space' on the left and 'Extracted 2D Principal Plane' on the right.

2. Formula
   Observation

   Left shows $x \in \mathbb{R}^3$ and [x_nose, x_eye, x_jaw]^T; right shows $z \in \mathbb{R}^2$ and [$z_1$, $z_2$]^$T = [[x \cdot e_1],[x \cdot e_2]]$.

3. Caption evidence
   Observation

   Bottom banner reads 'Storage Reduction: -33.3% (3 floats -> 2 floats per face)' and an arrow says 'Structure Preserved'.

1. Left 3D scatter plot with tilted plane

2. Right 2D scatter plot with PC axes

3. Color-coded point clusters

4. Formula boxes

5. Storage-reduction banner

6. Structure-preserved arrow

1. The right panel populates with projected points after the left panel establishes the original data.

2. Text and formulas appear to define the coordinate transform.

3. A banner quantifies storage reduction.

1. Corresponding colors link the same subjects across both panels.

2. The left panel remains the original 3D space while the right panel remains the extracted 2D space.

The split-screen makes the abstraction concrete: the same dataset is shown once in 3D raw coordinates and once in 2D principal coordinates, with explicit formulas linking the representations.

1. Diagram
   Observation

   Left panel titled '3D Feature Space' shows gray original points; right panel titled '2D Compressed Space' shows yellow compressed points; blue arrows connect them to cyan reconstructed points.

2. Formula
   Observation

   Top-right formula reads $\hat{\mathbf{x}} = \mathbf{W}\mathbf{z} = z_1\mathbf{v}_1 + z_2\mathbf{v}_2$; lower-right label reads $\mathbf{z} \in \mathbb{R}^2 \rightarrow \hat{\mathbf{x}} \in \mathbb{R}^3$.

1. Gray original 3D points

2. Yellow 2D compressed points

3. Cyan reconstructed 3D points

4. Blue transfer arrows

5. Reconstruction formula box

1. Yellow points are linked by arrows to new cyan points in the 3D panel.

2. The reconstruction formula appears and remains on screen.

3. The visual emphasis shifts from compression to approximate recovery.

1. The 2D coordinate system on the right and 3D system on the left stay fixed.

2. Each arrow represents one compressed point being lifted back to 3D.

The animation illustrates the inverse map from compressed coordinates to an approximated 3D reconstruction, showing that one can move back into the original ambient space but only to the plane spanned by the retained components.

1. Animation
   Observation

   Animation shows yellow dots in a 2D space being mapped via curved lines to cyan dots in a 3D space, which then settle onto a flat green plane.

1. 2D Compressed Space with yellow points

2. 3D Feature Space with axes x1, x2, x3

3. Green 2D Subspace Slice

4. Vectors v1 and v2

1. Yellow points in 2D space are transformed into cyan points in 3D space.

2. The cyan points align perfectly on the green 2D plane.

1. The mathematical relationship $\hat{x} = Wz$ remains constant.

2. The green plane represents the span of v1 and v2.

This visual demonstrates that reconstructing data from a lower-dimensional PCA space forces all points to lie exactly on the subspace defined by the retained principal components, losing any original orthogonal depth.

1. Animation
   Observation

   Animation shows a 3D coordinate system with a blue plane. Cyan points float above and below the plane, connected by red vertical lines to their projections on the plane.

1. Blue 2D plane

2. Floating cyan 3D points

3. Red connecting whiskers

4. Yellow arrow indicating PC3 direction

1. Red lines appear connecting the floating points to the plane.

2. A yellow arrow emerges pointing vertically, labeled 'PC3 = Error Direction'.

1. The blue plane remains fixed as the projection subspace.

2. The distance represented by the red lines corresponds to the reconstruction error.

The red whiskers visually quantify the reconstruction error, showing that the error is strictly aligned with the direction of the dropped principal component (the missing axis).

1. Animation
   Observation

   Screen transitions to a summary view showing '3D Feature Space' on the left and '2D Optimal Projection Canvas' on the right, connected by an arrow labeled 'Perpendicular Shadow'.

1. 3D Feature Space box

2. 2D Optimal Projection Canvas box

3. Perpendicular Shadow arrow

4. Dimensionality Criterion formula

1. The video synthesizes previous concepts into a unified diagram.

2. Formulas for matrix multiplication and variance retention are displayed.

1. The core concept of orthogonal projection remains the central theme.

This final sequence consolidates the geometric intuition of PCA, linking the algebraic matrix operations to the visual concept of casting a perpendicular shadow to maximize variance and minimize lost depth.

1. Animation
   Observation

   The green-bordered 'Dimensionality Criterion' card flips horizontally to reveal a blue-bordered 'NEXT EPISODE PREVIEW' card.

1. Dimensionality Criterion card

2. Next Episode Preview card

1. The front face of the card disappears.

2. The back face of the card appears with new text and formulas.

1. The overall layout and dark background remain consistent.

Visually separates the conclusion of the current topic from the teaser of the next topic.

1. Audio
   Observation

   Narrator explicitly contrasts the actual data distribution with points scattering randomly 'like static on an old television'.

2. Formula
   Observation

   Text box 'Hypothesis: Uncorrelated Static' is shown and then replaced by 'Reality: Tilted Correlated Cloud'.

Assuming that multi-dimensional feature data will naturally distribute uniformly or randomly across all dimensions.

In reality, natural features are highly correlated, causing the data to cluster into elongated, tilted clouds with dominant variance orientations, which makes dimensionality reduction possible.

1. Audio
   Observation

   Narrator warns that taking a snapshot from an 'awkward vantage' causes subjects to overlap into an 'indecipherable blur'.

2. Animation
   Observation

   Visual shows red points clumping severely when projected onto a non-aligned horizontal plane.

Believing that reducing dimensions by simply dropping coordinates or projecting onto arbitrary planes retains the useful information of the dataset.

Projecting onto a suboptimal angle causes severe point clumping and hides valuable differences. The projection plane must be mathematically optimized (e.g., via PCA) to align with the data's maximum spread.

1. Audio
   Observation

   Narrator explicitly warns: 'Notice that we are not randomly discarding the third axis.'

One might think dimensionality reduction here simply throws away one original coordinate axis at random.

The video states that the method slides each point along a trajectory perpendicular to the chosen plane, discarding only the depth variation normal to that plane rather than arbitrarily deleting an axis.

1. Audio
   Observation

   Narrator uses the pressed-flower analogy: thickness vanishes, yet perimeter, petal arrangement, and internal boundaries stay visible.

2. Caption evidence
   Observation

   Card text reads 'Thickness -> 0 | Pattern Kept'.

Flattening a 3D object might be assumed to destroy all informative structure.

The analogy distinguishes removed thickness from retained planar pattern: the third-dimensional extent disappears, but the 2D arrangement of features can remain visible.

1. Audio
   Observation

   Narrator says compression inevitably discards information and then asks whether the journey can be reversed.

2. Formula
   Observation

   Reconstruction is written with $\hat{\mathbf{x}}$, not $\mathbf{x}$.

Because the video shows a map back to 3D, viewers may infer that the original measurements are perfectly recovered.

The clip presents reconstruction as an approximate lift $\hat{\mathbf{x}}$ built from retained components, after explicitly stating that compression discards information.

1. Audio
   Observation

   Narrator explicitly states: 'Notice what happens. Our reconstructed points do not magically recover their original heights.'

A common misunderstanding is that reconstructing data from PCA coordinates will perfectly restore the original high-dimensional data, including all its depths and heights.

The video clarifies that reconstructed points are strictly trapped on the exact lower-dimensional slice they were projected onto; they cannot recover information in the orthogonal directions that were discarded.

1. Audio
   Observation

   Narrator first explains finding the optimal plane via PCA, then concludes that dimensionality reduction is literally casting an orthogonal shadow onto such a plane.

The mathematical optimization of finding the principal components (PCA) provides the exact geometric subspace required to perform the orthogonal shadow casting (projection) without losing structural information.

1. Audio
   Observation

   Narrator links natural correlations directly to the formation of an elongated, tilted cloud with specific orientations stretching farther than others.

Understanding that feature correlations create a dominant variance orientation is the necessary prerequisite for knowing why PCA seeks the direction of maximum spread to define the projection plane.

1. Formula
   Observation

   The formula $d_\perp = \min_{\mathbf{p} \in \mathcal{P}}$ \|$\mathbf{x} - \mathbf{p}$\| is introduced while the perpendicular drop lines are on screen.

2. Audio
   Observation

   Narrator links the drop lines to the distance sacrificed during compression.

The geometric notion of orthogonal shadowing is quantified by the perpendicular-distance formula.

1. Diagram
   Observation

   After defining single-point drop distance, the video scales up to many points collapsing onto the plane.

2. Audio
   Observation

   Narrator moves from 'precise distance sacrificed' to 'the entire 3D cloud collapses in unison'.

The per-point loss measure is generalized to a whole dataset being flattened onto the 2D surface.

1. Audio
   Observation

   Narrator says: 'To utilize this mathematically, we extract the plane into its own independent two-dimensional coordinate system.'

2. Formula
   Observation

   The side-by-side panel replaces the geometric plane picture with explicit coordinates $z_1$,$z_2$.

The coordinate-compression method is the algebraic implementation of the previously visualized collapse onto the plane.

1. Caption evidence
   Observation

   The storage banner appears directly under the side-by-side compression display.

2. Audio
   Observation

   Narrator ties the reduction from three numbers to two numbers to a 33% storage decrease.

The storage-reduction claim is a direct numerical consequence of representing each sample with two coordinates instead of three.

1. Audio
   Observation

   Narrator asks whether the journey can be reversed after noting that compression discards information.

2. Formula
   Observation

   The reconstruction formula $\hat{\mathbf{x}} = \mathbf{W}\mathbf{z}$ uses the same $\mathbf{z}$ introduced in the compression step.

Reconstruction depends on the compressed coordinate representation produced by the earlier compression step.

1. Diagram
   Observation

   First half shows points dropped to the plane; last half shows points lifted from the plane back into 3D.

2. Audio
   Observation

   Narration contrasts forward compression with reversing the journey.

Orthogonal shadowing maps 3D to the plane, while reconstruction maps plane coordinates back to an approximated 3D location.

1. Audio
   Observation

   Narrator connects the trapped points on the 2D slice to the gap between the original and reconstructed points, defining it as reconstruction error.

The mathematical formula for reconstruction dictates that points fall on the subspace, which directly leads to the geometric interpretation that the reconstruction error aligns with the dropped principal components.

1. Audio
   Observation

   Narrator transitions from discussing the physical thickness of the error to the master insight about what 95% variance explained actually means for the data's dimensionality.

The video contrasts the physical reality of reconstruction error (lost depth) with the statistical measure of variance explained, showing that high variance retention implies negligible reconstruction error.

1. Formula
   Observation

   The final formula mathematically formalizes the concept of retained variance discussed earlier.

The intuitive explanation of 95% variance meaning the data is barely 3D is formally equivalent to the mathematical criterion of Retained Essence calculated via eigenvalues.

1. Audio
   Observation

   Subscribe because next we explore matrix calculus, gradients of vectors.

The video transitions from the statistical concept of PCA to the mathematical tool of gradients, implying that matrix calculus and gradients will be used in subsequent topics related to optimization or advanced dimensionality reduction techniques.

1. Audio
   Observation

   Narrator mentions finding the 'next widest perpendicular direction'.

1. Defining the Optimal Projection Plane via Principal Components

1. Animation
   Observation

   Visual shows red points overlapping into a blur on a horizontal plane.

1. Shadow Projection Analogy for Dimensionality Reduction

1. Audio
   Observation

   Narrator explains that natural human facial proportions correlate with each other.

1. Geometry of Correlated Feature Clouds

1. Audio
   Observation

   Narrator explains points travel perpendicular to the chosen plane until they land flat on the surface.

2. Formula
   Observation

   Formula box defines $d_\perp$.

1. Orthogonal shadow projection onto a plane
2. Perpendicular distance as compression loss

1. Formula
   Observation

   Displayed equation $d_\perp = \min_{\mathbf{p} \in \mathcal{P}}$ \|$\mathbf{x} - \mathbf{p}$\|.

2. Audio
   Observation

   Narrator calls this the precise distance sacrificed during compression.

1. Perpendicular distance as compression loss

1. Audio
   Observation

   Narrator explicitly says they are not randomly discarding the third axis.

1. Orthogonal shadow projection onto a plane
2. Mistaking PCA projection for arbitrary axis deletion

1. Caption evidence
   Observation

   Card reads 'Analogy: Pressed Flower', '3D Volume -> Planar Pattern', 'Thickness -> 0 | Pattern Kept'.

2. Audio
   Observation

   Narrator explains thickness vanishes while perimeter and internal boundaries remain visible.

1. Collapse of a 3D cloud onto a 2D surface
2. Confusing loss of thickness with loss of all structure

1. Formula
   Observation

   Right panel shows [$z_1$,$z_2$]^$T = [[x \cdot e_1],[x \cdot e_2]]$.

2. Audio
   Observation

   Narrator says each face needs its position along PC1 and PC2.

1. Side-by-side coordinate compression from 3 features to 2 PCs

1. Caption evidence
   Observation

   Banner states 'Storage Reduction: -33.3% (3 floats -> 2 floats per face)'.

2. Audio
   Observation

   Narrator says storage is reduced by thirty-three percent.

1. Storage reduction from 3 floats to 2 floats per face
2. Face-measurement compression example

1. Audio
   Observation

   Narrator asks whether the journey can be reversed and then describes lifting points back through principal component directions.

2. Formula
   Observation

   Formula shows $\hat{\mathbf{x}} = \mathbf{W}\mathbf{z} = z_1\mathbf{v}_1 + z_2\mathbf{v}_2$.

1. Lifting 2D coordinates back to 3D
2. Assuming reconstruction restores the exact original point

1. Formula
   Observation

   Reconstruction uses $\hat{\mathbf{x}}$ rather than $\mathbf{x}$.

2. Audio
   Observation

   Narrator frames this as rebuilding original measurements after compression discarded information.

1. Lifting 2D coordinates back to 3D
2. Assuming reconstruction restores the exact original point

1. Audio
   Observation

   Narrator discusses the limitations of lifting shadows back to 3D.

1. PCA Reconstruction Formula
2. Misconception about PCA Reconstruction