No One Taught SVD (Singular Value Decomposition) Like This

1. Formula
   Observation

   The matrix A is displayed as a 2x2 matrix with entries 4, -2, 2, and 1.

A

A 2x2 matrix representing a linear transformation.

2x2 real matrices

1. Formula
   Observation

   The vector $v_1$ is initially shown as [0.8, 0.6]^T and later updated to [0.334, 0.942]^T and [-0.336, -0.942]^T.

$v_1$

A unit vector in the domain space being transformed by matrix A.

Unit vectors in $R^2$

1. Formula
   Observation

   The equation Av_1 = $y_1$ is shown, and $y_1$ is calculated as [2, 2.2]^T for the initial $v_1$.

$y_1$

The transformed vector resulting from multiplying matrix A by the unit vector $v_1$.

Vectors in $R^2$

1. Formula
   Observation

   The vector $v_2$ is introduced and later shown as [-0.942, 0.334]^T and [0.942, -0.336]^T.

$v_2$

A second unit vector in the domain space, chosen to be orthogonal to $v_1$.

Unit vectors in $R^2$

1. Formula
   Observation

   The equation Av_2 = $y_2$ is shown.

$y_2$

The transformed vector resulting from multiplying matrix A by the unit vector $v_2$.

Vectors in $R^2$

1. Formula
   Observation

   The scalar sigma is introduced in the equation y = sigma u.

sigma

A scalar value representing the magnitude of the transformed vector y.

Real numbers

1. Formula
   Observation

   Matrix $A = [[4, -2], [2, 1]]$ is displayed on the left side of the screen.

A

The original 2x2 matrix being decomposed.

2x2 real matrix

1. Formula
   Observation

   Vector $v_1 = [0.334, 0.942]^T$ is shown in blue.

$v_1$

First right singular vector.

Unit vector in $R^2$

1. Formula
   Observation

   Vector $v_2 = [-0.942, 0.334]^T$ is shown in red.

$v_2$

Second right singular vector.

Unit vector in $R^2$

1. Formula
   Observation

   $y_1$ = Av_1 = [-0.548, 1.61]^T is calculated and displayed.

$y_1$

Transformed vector corresponding to $v_1$.

Vector in $R^2$

1. Formula
   Observation

   $y_2$ = Av_2 = [-4.436, -1.55]^T is calculated and displayed.

$y_2$

Transformed vector corresponding to $v_2$.

Vector in $R^2$

1. Formula
   Observation

   ||$y_1$|| = sigma_1 = 1.7 is displayed in red.

sigma_1

First singular value, magnitude of $y_1$.

Positive real scalar

1. Audio
   Observation

   The narrator states the goal is to find an orthogonal set of vectors that remain orthogonal when transformed by A.

2. Formula
   Observation

   The word 'orthogonal' appears on screen next to the input vectors and output vectors.

An orthogonal set of vectors is a collection of vectors that are mutually perpendicular. In the context of Singular Value Decomposition, the goal is to find a specific set of orthogonal unit vectors in the domain such that when they are transformed by a matrix A, their images in the codomain are also orthogonal.

1. The vectors in the domain must be unit vectors.

2. The transformation is defined by a matrix A.

1. Audio
   Observation

   The narrator explains that since $v_1$ and $v_2$ are unit vectors but $y_1$ and $y_2$ are not, y can be represented using a scalar sigma acting as the magnitude.

When a unit vector v is transformed by a matrix A into a vector y, the resulting vector y is generally not a unit vector. The length or magnitude of this transformed vector y is called the singular value, denoted by sigma. It represents how much the matrix A stretched or squashed the original unit vector.

1. The input vector v must be a unit vector.

2. The output vector y is the result of the matrix-vector multiplication Av.

1. Orthogonal Set under Linear Transformation

1. Audio
   Observation

   Narrator explains that SVD breaks down a matrix into three simple jobs: rotation, stretching, and rotation.

2. Formula
   Observation

   $A = U \Sigma V^T$ is derived step-by-step.

SVD factorizes a matrix A into $U \Sigma V^T$, where U and V are orthogonal matrices representing rotations/reflections, and $\Sigma$ is a diagonal matrix representing stretching along principal axes.

1. A is a real matrix

2. U and V are orthogonal

3. $\Sigma$ has non-negative diagonal entries

1. Orthogonal Matrix Property

1. Audio
   Observation

   Narrator states 'Since V is an orthogonal matrix... V inverse is the same as V transpose'.

An orthogonal matrix has mutually orthogonal column vectors of unit length. Its inverse equals its transpose.

1. Columns of V are orthonormal

1. Formula
   Observation

   Max ||y|| subject to ||v|| = 1 is shown.

2. Audio
   Observation

   Narrator explains SVD solves this optimization problem to find maximum stretch.

SVD identifies the unit vector v that maximizes the norm of the transformed vector Av. The maximum value is the largest singular value.

1. v is a unit vector

1. Singular Value Decomposition Definition

1. Audio
   Observation

   SVD is used for image compression and denoising by reducing dimensionality and preserving significant singular values.

2. Formula
   Observation

   $A = UΣV^T$ is shown, and the zero column in Σ is highlighted and discarded.

SVD compresses data by discarding columns in the Sigma matrix that have zero or very small singular values, as they do not contribute significantly to the output. The remaining columns with the largest singular values capture most of the meaningful information.

1. Sigma matrix is arranged in descending order of singular values.

1. Audio
   Observation

   The narrator introduces Principal Component Analysis (PCA) as finding directions in which data spreads out.

2. Diagram
   Observation

   A scatter plot with PC1 and PC2 axes is shown.

PCA is a technique used to identify the directions (principal components) in which the data varies the most. It can be solved using SVD.

1. SVD for Image Compression

1. Formula
   Observation

   AV = $U\Sigma$ is written on screen.

2. Audio
   Observation

   Narrator says 'you can see that AV equals U times Sigma'.

For the constructed matrices U, $\Sigma$, and V, the equation AV = $U\Sigma$ holds.

1. A, U, $\Sigma$, V are defined as in the SVD derivation

Universal for the specific example matrices

1. Audio
   Observation

   The narrator says, 'Now let us do some algebra and see some magic... We have these two equations with us, right? $v_1$ and $v_2$ are already unit vectors, but $y_1$ and $y_2$ are not. So we can represent it using some scalar value sigma which acts as the magnitude and the unit...'

2. Formula
   Observation

   The equations Av_1 = $y_1$ and Av_2 = $y_2$ are grouped together, and then the expression y = sigma u is written.

1. Expression

   $$Av_1 = y_1 \quad \text{and} \quad Av_2 = y_2$$
   Explanation

   Start with the given transformation equations for the two orthogonal unit vectors.

   Justification

   Definition of the linear transformation applied to the basis vectors.

   Shown in the video
2. Expression

   $$\|v_1\| = 1, \quad \|v_2\| = 1$$
   Explanation

   Note that the input vectors are unit vectors by construction.

   Justification

   Stated explicitly in the audio and shown in the visual setup.

   Shown in the video
3. Expression

   $$\|y_1\| \neq 1, \quad \|y_2\| \neq 1$$
   Explanation

   Observe that the output vectors are generally not unit vectors because the matrix A scales them.

   Justification

   Visual evidence shows the red and maroon vectors are longer than the blue input vectors.

   Derived from the video
4. Expression

   $$y = \sigma u$$
   Explanation

   Express each transformed vector y as the product of its magnitude (singular value) sigma and a new unit vector u indicating its direction.

   Justification

   Standard vector decomposition into magnitude and direction.

   Shown in the video

The transformed vectors $y_1$ and $y_2$ can be factored into a scalar magnitude (singular value) and a unit direction vector, setting up the core structure of the Singular Value Decomposition.

1. Formula
   Observation

   Step-by-step algebraic manipulation from $A[v_1 v_2] = [\sigma_1 u_1 \sigma_2 u_2]$ to $A = U\Sigma V^T$.

1. Expression

   $$A [v_1 \quad v_2] = [\sigma_1 u_1 \quad \sigma_2 u_2]$$
   Explanation

   Combine the individual transformations Av_i = $\sigma_i u_i$ into a single matrix equation.

   Justification

   Definition of matrix-vector multiplication and block matrices.

   Shown in the video
2. Expression

   $$A [v_1 \quad v_2] = [u_1 \quad u_2] \begin{bmatrix} \sigma_1 & 0 \\ 0 & \sigma_2 \end{bmatrix}$$
   Explanation

   Factor the right side into a matrix of u vectors and a diagonal sigma matrix.

   Justification

   Matrix multiplication properties.

   Shown in the video
3. Expression

   $$AV = U\Sigma$$
   Explanation

   Substitute $V = [v_1 v_2]$, $U = [u_1 u_2]$, and $\Sigma$ = diag($\sigma_1$, $\sigma_2$).

   Justification

   Definition of matrices U, V, $\Sigma$.

   Shown in the video
4. Expression

   $$(AV)V^{-1} = U\Sigma V^{-1}$$
   Explanation

   Multiply both sides by V inverse on the right.

   Justification

   Algebraic manipulation.

   Shown in the video
5. Expression

   $$A = U\Sigma V^T$$
   Explanation

   Use the property that for orthogonal V, $V^{-1} = V^T$.

   Justification

   Property of orthogonal matrices.

   Shown in the video

The matrix A can be decomposed as $A = U\Sigma V^T$.

1. Audio
   Observation

   We find that the last column of sigma is zero vector, and thus we simply discard the columns where the sigma values become zero because they do not contribute anything to the output.

2. Formula
   Observation

   The third column of Σ is [0, 0]^T and is crossed out.

1. Expression

   $$Σ = [5, 0, 0; 0, 3, 0]$$
   Explanation

   Identify the zero column in the Sigma matrix.

   Justification

   Visual inspection of the provided Sigma matrix.

   Shown in the video
2. Expression

   $$Discard 3rd column of Σ and corresponding rows of V^T$$
   Explanation

   Remove the dimensions associated with zero singular values.

   Justification

   They do not contribute to the matrix multiplication output.

   Shown in the video

Zero singular values correspond to dimensions that can be safely removed without losing information.

1. Formula
   Observation

   $A = [[4, -2], [2, 1]]$, $v_1 = [[0.8], [0.6]]$, Av_1 = $y_1 = [[2], [2.2]]$

2. Animation
   Observation

   A dark blue vector rotates, and a red vector transforms accordingly.

Given the matrix $A = [[4, -2], [2, 1]]$ and a unit vector $v_1 = [[0.8], [0.6]]$, find the transformed vector $y_1$.

1. Matrix $A = [[4, -2], [2, 1]]$

2. Unit vector $v_1 = [[0.8], [0.6]]$

Calculate the resulting vector $y_1$ after applying the transformation A to $v_1$.

1. Expression

   $$y_1 = A v_1 = \begin{bmatrix} 4 & -2 \\ 2 & 1 \end{bmatrix} \begin{bmatrix} 0.8 \\ 0.6 \end{bmatrix}$$
   Explanation

   Set up the matrix-vector multiplication.

   Justification

   Definition of linear transformation.

   Shown in the video
2. Expression

   $$y_1 = \begin{bmatrix} (4)(0.8) + (-2)(0.6) \\ (2)(0.8) + (1)(0.6) \end{bmatrix} = \begin{bmatrix} 3.2 - 1.2 \\ 1.6 + 0.6 \end{bmatrix} = \begin{bmatrix} 2 \\ 2.2 \end{bmatrix}$$
   Explanation

   Perform the arithmetic to find the components of the transformed vector.

   Justification

   Matrix multiplication rules.

   Derived from the video

The transformed vector is $y_1 = [[2], [2.2]]$.

The visual animation shows the red vector pointing to approximately (2, 2.2) on the coordinate plane.

1. Formula
   Observation

   Specific numerical values for A, $v_i$, $y_i$, $u_i$, $\sigma_i$ are shown throughout.

Find the SVD of the matrix $A = [[4, -2], [2, 1]]$.

1. Matrix $A = [[4, -2], [2, 1]]$

2. Right singular vectors $v_1 = [0.334, 0.942]^T$, $v_2 = [-0.942, 0.334]^T$

Determine matrices U, $\Sigma$, and V such that $A = U\Sigma V^T$.

1. Expression

   $$y_1 = Av_1 = [-0.548, 1.61]^T$$
   Explanation

   Compute the transformation of $v_1$ by A.

   Justification

   Matrix-vector multiplication.

   Shown in the video
2. Expression

   $$\sigma_1 = ||y_1|| = 1.7$$
   Explanation

   Calculate the magnitude of $y_1$ to get the first singular value.

   Justification

   Definition of singular value.

   Shown in the video
3. Expression

   $$u_1 = y_1 / \sigma_1 = [-0.322, 0.947]^T$$
   Explanation

   Normalize $y_1$ to get the first left singular vector.

   Justification

   Definition of unit vector.

   Shown in the video
4. Expression

   $$y_2 = Av_2 = [-4.436, -1.55]^T$$
   Explanation

   Compute the transformation of $v_2$ by A.

   Justification

   Matrix-vector multiplication.

   Shown in the video
5. Expression

   $$\sigma_2 = ||y_2|| = 4.7$$
   Explanation

   Calculate the magnitude of $y_2$ to get the second singular value.

   Justification

   Definition of singular value.

   Shown in the video
6. Expression

   $$u_2 = y_2 / \sigma_2 = [-0.944, -0.33]^T$$
   Explanation

   Normalize $y_2$ to get the second left singular vector.

   Justification

   Definition of unit vector.

   Shown in the video
7. Expression

   $$V = [[0.334, -0.942], [0.942, 0.334]]$$
   Explanation

   Construct matrix V from columns $v_1$ and $v_2$.

   Justification

   Definition of V in SVD.

   Shown in the video
8. Expression

   $$U = [[-0.322, -0.944], [0.947, -0.33]]$$
   Explanation

   Construct matrix U from columns $u_1$ and $u_2$.

   Justification

   Definition of U in SVD.

   Shown in the video
9. Expression

   $$\Sigma = [[1.7, 0], [0, 4.7]]$$
   Explanation

   Construct diagonal matrix $\Sigma$ from singular values.

   Justification

   Definition of $\Sigma$ in SVD.

   Shown in the video

$A = U\Sigma V^T$ with the calculated U, $\Sigma$, and V.

Visual confirmation of matrix assembly and final equation $A = U\Sigma V^T$.

1. Formula
   Observation

   Matrices A, U, Σ, and $V^T$ are explicitly given.

Compress the matrix A = [3, 2, 2; 2, 3, -2] using its SVD.

1. A = [3, 2, 2; 2, 3, -2]

2. U = [1/√2, 1/√2; 1/√2, -1/√2]

3. Σ = [5, 0, 0; 0, 3, 0]

4. $V^T$ = [1/√2, 1/√2, 0; 1/√18, -1/√18, 4/√18; $2/3$, -2/3, $1/3$]

Find the compressed representation by discarding zero singular values.

1. ExpressionIdentify zero column in Σ
   Explanation

   The third column of Σ is [0, 0]^T.

   Justification

   Direct observation of the matrix.

   Shown in the video
2. ExpressionKeep first two columns
   Explanation

   Retain the columns of U and the corresponding rows of $V^T$ associated with the non-zero singular values 5 and 3.

   Justification

   These carry the meaningful information.

   Shown in the video

The compressed representation uses only the first two singular values (5 and 3) and their corresponding vectors.

The discarded column had a zero singular value, meaning its contribution to the product $UΣV^T$ was exactly zero.

1. Diagram
   Observation

   A grid of images showing an original plant photo and its reconstructions with varying k values.

Reconstruct an image using only the first k singular values.

1. Original image matrix ($k=3648$)

2. SVD of the image matrix

Observe the effect of varying k on image quality.

1. Expression

   $$k = 1$$
   Explanation

   Reconstruct image using only the largest singular value.

   Justification

   Captures the most basic structure, resulting in a blurry image.

   Shown in the video
2. Expression

   $$k = 5, 10, 20$$
   Explanation

   Gradually increase k to include more singular values.

   Justification

   More details and sharper features appear.

   Shown in the video
3. Expression

   $$k = 200$$
   Explanation

   Use the first 200 components.

   Justification

   Produces a sharp, clear image, showing that most important structure is captured by a small fraction of the total components.

   Shown in the video

Increasing k improves image quality, with $k=200$ providing a visually lossless reconstruction compared to the original $k=3648$.

Visual comparison of the reconstructed images against the original.

1. Animation
   Observation

   The dark blue vector $v_1$ rotates around the origin, and the red vector $y_1$ changes its direction and length correspondingly.

1. Dark blue vector $v_1$

2. Red vector $y_1$

3. Coordinate axes

1. The angle of $v_1$ increases continuously.

2. The direction and magnitude of $y_1$ change non-uniformly as $v_1$ rotates.

1. The relationship $y_1 = A v_1$ is maintained throughout the animation.

2. The length of $v_1$ remains constant at 1.

This visualizes how a general linear transformation stretches, squashes, and rotates vectors differently depending on their initial direction.

1. Animation
   Observation

   Two blue vectors $v_1$ and $v_2$ rotate together maintaining a 90-degree angle. Two corresponding output vectors $y_1$ (red) and $y_2$ (maroon) rotate, and an angle indicator between them fluctuates, occasionally hitting exactly 90 degrees.

1. Blue vectors $v_1$ and $v_2$

2. Red vector $y_1$

3. Maroon vector $y_2$

4. Angle indicator

1. The pair ($v_1$, $v_2$) rotates.

2. The angle between ($y_1$, $y_2$) changes dynamically.

3. Specific orientations are highlighted where the angle between $y_1$ and $y_2$ is exactly 90 degrees.

1. The angle between $v_1$ and $v_2$ is always 90 degrees.

2. The lengths of $v_1$ and $v_2$ are always 1.

This demonstrates the core search in SVD: finding a specific orientation of orthogonal unit vectors in the domain whose images under the transformation are also orthogonal.

1. Animation
   Observation

   A blue unit vector rotates around the origin, and a red vector (its image under A) changes length and direction.

1. Blue unit vector v

2. Red transformed vector y=Av

3. Coordinate axes

1. Angle of v changes continuously

2. Length and angle of y change as v rotates

1. Length of v remains 1

2. Relationship y=Av is maintained

Demonstrates how the linear transformation A stretches and rotates different input directions by varying amounts.

1. Diagram
   Observation

   Vectors $v_1$, $v_2$, $u_1$, $u_2$ are plotted with lengths corresponding to singular values.

1. Vectors $v_1$, $v_2$ (blue)

2. Vectors $u_1$, $u_2$ (red/purple)

3. Labels $\sigma_1$, $\sigma_2$

1. Orthogonality of $v_1$, $v_2$ and $u_1$, $u_2$

Shows the principal axes of the transformation and the scaling factors (singular values) along those axes.

1. Animation
   Observation

   An orange circle highlights the zero column in Σ, followed by a red cross over it. Then blue boxes highlight the non-zero columns.

1. Matrix Σ

2. Orange circle

3. Red cross

4. Blue boxes

1. Zero column is highlighted and crossed out.

2. Non-zero columns are boxed.

1. Matrices U and $V^T$ remain unchanged during this specific animation step.

Visually demonstrates the process of identifying and discarding dimensions with zero variance or contribution.

1. Diagram
   Observation

   A 3x3 grid of images appears, showing progressive reconstruction of a plant photo with increasing k values.

1. Original image

2. Reconstructed images for $k=1$, 5, 10, 20, 50, 100, 200

1. Images become progressively sharper and more detailed as k increases.

1. The subject matter (plant) remains the same across all images.

Illustrates the trade-off between the number of retained singular values (compression level) and the fidelity of the reconstructed data.

1. Diagram
   Observation

   A scatter plot with red 'x' marks and two green arrows labeled PC1 and PC2.

1. Data points (red x)

2. Principal Component 1 axis (green arrow)

3. Principal Component 2 axis (green arrow)

1. None, static diagram.

1. The orientation of the principal components relative to the data spread.

Shows how PCA finds new orthogonal axes (PC1, PC2) that maximize the variance of the projected data points.

1. Audio
   Observation

   Narrator mentions that many videos show how to find V, Sigma, U but hardly explain what they mean.

Learning SVD only as a computational algorithm without understanding the geometric roles of U, $\Sigma$, and V.

U and V represent rotations/reflections (change of basis), while $\Sigma$ represents pure stretching along the new axes.

1. Audio
   Observation

   The narrator connects finding orthogonal sets to the algebraic representation using sigma.

Finding the specific orthogonal input vectors ($v_1$, $v_2$) is the necessary first step before their transformed outputs ($y_1$, $y_2$) can be cleanly decomposed into magnitudes (singular values) and orthogonal directions.

1. Audio
   Observation

   Narrator uses the property of orthogonal matrices to simplify V inverse to V transpose.

The derivation of the final SVD formula relies on the property that the inverse of an orthogonal matrix is its transpose.

1. Formula
   Observation

   Optimization problem Max ||Av|| is linked to the first column of U and largest sigma.

Finding the SVD is equivalent to solving the optimization problem of maximizing the stretch of a unit vector.

1. Audio
   Observation

   The narrator states that figuring out the directions in which data spreads out is called PCA, and it can be solved using SVD.

SVD is a mathematical tool used to compute the Principal Component Analysis (PCA) of a dataset.

1. Audio
   Observation

   Narrator introduces sigma as the magnitude of the non-unit transformed vectors.

1. Singular Value as Magnitude

1. Audio
   Observation

   Narrator explains the goal of finding orthogonal vectors that remain orthogonal after transformation.

1. Orthogonal Set under Linear Transformation

1. Audio
   Observation

   Entire video focuses on explaining the meaning of SVD components.

1. Singular Value Decomposition Definition
2. Geometric Interpretation Diagram

1. Formula
   Observation

   Optimization formulation is presented.

1. SVD as Optimization Problem

1. Audio
   Observation

   Explanation of how SVD discards zero/small singular values for compression.

1. SVD for Image Compression
2. Discarding Zero Singular Values
3. Compressing a 2x3 Matrix

1. Diagram
   Observation

   Grid of images showing reconstruction quality vs k.

1. Image Compression via SVD
2. Image Reconstruction Grid

1. Audio
   Observation

   Definition of PCA as finding directions of data spread.

1. Principal Component Analysis (PCA)
2. PCA Scatter Plot
3. s330-cr-svd-pca