Condition Numbers

1. Formula
   Observation

   Slide shows the equation $Ax=b$.

2. Audio
   Observation

   Speaker says, "Let's say we want to solve for Ax is b, which is just linear regression here."

A

Matrix in the linear system $Ax=b$, identified by the speaker as the linear-regression setting.

Video does not state dimensions or rank conditions explicitly.

1. Formula
   Observation

   Slide shows $Ax=b$ and later $\|\delta x\|$ in the condition-number formula.

2. Audio
   Observation

   Speaker refers to changing $x$ when explaining the condition number.

x

Unknown vector in the linear system; perturbations of it are denoted $\delta x$.

Video does not state dimension explicitly.

1. Formula
   Observation

   Slide shows the equation $Ax=b$.

b

Right-hand side vector of the linear system.

Video does not state dimension explicitly.

1. Formula
   Observation

   Slide title reads "Ill conditioned problem: large or infinite condition number" and displays $\kappa$ and $\kappa(A)$.

2. Audio
   Observation

   Speaker introduces "the notion of condition number that quantifies how hard and how undefined a problem is."

$\kappa$

Condition number, used to quantify how hard or undefined a problem is.

Defined on the slide for the linear system context.

1. Formula
   Observation

   Slide shows $\lim_{\varepsilon \to 0}$ in the definition of $\kappa$.

2. Audio
   Observation

   Speaker says the condition number is defined as the limit of epsilon as we make it small.

$\varepsilon$

Small positive perturbation radius tending to zero in the supremum definition.

Appears under the limit $\varepsilon \to 0$ and constraint $\|\delta x\| \le \varepsilon$.

1. Formula
   Observation

   Slide shows $\|\delta x\| \le \varepsilon$ and $\frac{\|\delta f\|}{\|\delta x\|}$.

2. Audio
   Observation

   Speaker describes looking at the ratio of changes within a small ring around a value.

$\delta x$

Perturbation of the input/solution variable $x$.

Constrained by $\|\delta x\| \le \varepsilon$ in the displayed formula.

1. Formula
   Observation

   Slide shows $\frac{\|\delta f\|}{\|\delta x\|}$.

2. Audio
   Observation

   Speaker mentions "the ratio of the change in f".

1. The slide defines the ratio using $\delta f$, but the video does not explicitly define $f$ in terms of $A$, $x$, or $b$ within this clip.

$\delta f$

Perturbation/change in the output quantity $f$ appearing in the condition-number ratio.

Used only inside the displayed ratio $\frac{\|\delta f\|}{\|\delta x\|}$.

1. Formula
   Observation

   Slide shows $\kappa(A)=\frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$.

1. The video does not verbally define singular values in this clip.

$\sigma_{\max(A)}$

Largest singular value of $A$ appearing in the formula for $\kappa(A)$.

Defined by the displayed formula only.

1. Formula
   Observation

   Slide shows $\kappa(A)=\frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$.

1. The video does not verbally define singular values in this clip.

$\sigma_{\min(A)}$

Smallest singular value of $A$ appearing in the denominator of $\kappa(A)$.

Defined by the displayed formula only.

1. Formula
   Observation

   Slide shows the equation Ax = b.

Ax = b

Linear system with matrix A and vectors x and b.

Matrix A is a linear operator; x and b are vectors in compatible spaces.

1. Formula
   Observation

   Slide shows $κ = \lim _{ε\to 0}$ sup_{||δx||≤ε} ||δf|| / ||δx||.

2. Audio
   Observation

   The lecturer says the condition number measures how small changes in x produce changes in f.

κ

Condition number of the problem, defined as the limiting worst-case relative output change per unit relative input perturbation.

Nonnegative scalar; may be infinite for ill-conditioned or singular problems.

1. Formula
   Observation

   Formula contains δx in the supremum constraint and denominator.

2. Audio
   Observation

   The lecturer refers to small changes in x that can happen by noise.

δx

Perturbation of the input variable x.

Vector perturbation with norm ||δx|| ≤ ε.

1. Caption evidence
   Observation

   Slide text: "If we are overcomplete" and "linear regression is well defined".

2. Audio
   Observation

   Speaker says that if we have more data, then we span all the dimensions, linear regression is well defined, and we can solve it with matrix inversion or gradient descent with the same solution.

In the first slide, the lecturer states that when the data situation is "overcomplete," linear regression is well defined. He further explains that in this case one can solve it either by matrix inversion or by gradient descent, and the resulting solution is the same.

1. The data situation is described as overcomplete.

2. The speaker additionally says there are more data and that all dimensions are spanned.

1. Caption evidence
   Observation

   Slide text: "If we are less than complete" and "it is undefined (and thus throws an error)".

2. Audio
   Observation

   Speaker says that if we have less data and do not have enough data, the answer is undefined and will throw an error depending on implementation, though most implementations will throw an error.

The lecturer contrasts the previous case with a "less than complete" situation. Here he says the answer is undefined and therefore throws an error. He adds that the exact behavior depends on implementation, but most implementations will throw an error.

1. The data situation is described as less than complete.

2. The speaker also frames this as not having enough data.

1. Audio
   Observation

   Speaker says, "how we solve that problem, like the mean squared error, we can still make it to go small. It's just that there's no longer one solution. All of a sudden, there's lots of solutions to that."

1. This statement is spoken while the first slide remains on screen; the clip does not show a separate formula for mean squared error.

Before introducing condition numbers, the lecturer notes that even when the problem is not uniquely solvable, the mean squared error can still be driven small. The key change is uniqueness: instead of one solution, there are many solutions.

1. Applies to the problematic case discussed immediately before the transition to ill-conditioned problems.

1. Less-than-complete case makes the answer undefined

1. Caption evidence
   Observation

   Slide title: "Ill conditioned problem: large or infinite condition number".

2. Audio
   Observation

   Speaker says there is a way of thinking more precisely about such problems and introduces the notion of condition number that quantifies how hard and how undefined a problem is.

On the second slide, the lecturer introduces the condition number as a precise way to think about ill-conditioned problems. He describes it as quantifying how hard and how undefined a problem is.

1. Introduced in the context of ill-conditioned problems.

1. Formula
   Observation

   Slide shows $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$.

2. Audio
   Observation

   Speaker says the condition number is defined as the limit of epsilon as we make it small of the largest value in that area of the ratio of the change in f, and describes looking within a small ring for the steepest area.

1. The spoken explanation uses informal geometric language (ring, steepest area) rather than restating the full symbolic expression verbatim.

The slide gives the condition number as a limiting supremum of the ratio of output perturbation to input perturbation over all perturbations whose size is at most $\varepsilon$, as $\varepsilon$ tends to zero. The lecturer interprets this verbally as looking in a shrinking neighborhood for the largest such ratio, i.e. the steepest direction of change.

1. Uses the linear-system setting introduced on the same slide.

2. The formula is displayed explicitly; the clip does not add further hypotheses verbally.

1. Condition number quantifies hardness and indefiniteness

1. Formula
   Observation

   Slide shows $\kappa(A) = \frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$.

1. The video does not explain singular values or prove this identity within the clip.

The same slide also gives a matrix-specific expression for the condition number: $\kappa(A)$ equals the largest singular value of $A$ divided by the smallest singular value of $A$.

1. Stated for the matrix $A$ in the displayed linear-system context.

1. Supremum-limit definition of the condition number

1. Caption evidence
   Observation

   Slide title reads 'Ill conditioned problem: large or infinite condition number'.

2. Audio
   Observation

   The lecturer explains that the condition number measures how difficult the problem is.

An ill-conditioned problem is one whose condition number is large or infinite, meaning very small input perturbations can cause disproportionately large output changes.

1. Applies to problems where an input perturbation δx induces an output perturbation δf.

2. The slide specifically frames this through Ax=b and $κ(A)$.

1. General condition number formula
2. Condition number of a matrix via singular values

1. Formula
   Observation

   Slide displays $κ = \lim _{ε\to 0}$ sup_{||δx||≤ε} ||δf|| / ||δx||.

2. Audio
   Observation

   The lecturer says we ask how small changes in x, which can happen by noise, produce changes in f.

The general condition number is the limiting worst-case amplification factor from input perturbation to output perturbation. It asks how much f can change relative to a small change in x.

1. Uses norms on input and output perturbations.

2. The supremum is taken over perturbations with ||δx|| ≤ ε before letting $ε\to 0$.

1. Formula
   Observation

   Slide displays $κ(A) = σ$_max(A) / σ_min(A).

2. Audio
   Observation

   The lecturer says, 'What is the condition number? Is the largest singular value of the matrix A divided by the smallest singular value.'

For the linear system Ax=b, the condition number of A is given by the ratio of the largest singular value to the smallest singular value.

1. Stated for the linear-equation case Ax=b.

2. If σ_min(A)=0, the displayed ratio indicates an infinite condition number.

1. General condition number formula

1. Audio
   Observation

   The lecturer says there are dimensions where moving causes a really big effect and other dimensions where it is really small.

2. Audio
   Observation

   He also says we are asking what dimension is most fragile and has the most effect.

The lecturer interprets conditioning geometrically: some directions in input space amplify the output strongly, while others barely affect it. The condition number captures the worst such amplification.

1. Presented as intuition rather than a formal proof.

2. Tied to the singular-value formula for Ax=b.

1. Condition number of a matrix via singular values

1. Caption evidence
   Observation

   Second slide states 'We usually have more parameters than data, so $κ = \infty$' and 'We are generally overparameterized'.

2. Audio
   Observation

   The lecturer says deep learning is weird because in many cases we have more parameters than data.

In the deep-learning setting discussed here, having more parameters than data is summarized on the slide as $κ = \infty$, indicating an extremely ill-conditioned inverse problem.

1. This is the slide's stated characterization for typical deep-learning regression settings.

2. The lecturer adds that data points often lie in relatively low-dimensional structure.

1. Ill-conditioned problem
2. General condition number formula

1. Caption evidence
   Observation

   Slide text reads 'So regression generally depends on how we solve the problem!' and 'This is *the* key to understand DL successes'.

When the problem is overparameterized and $κ = \infty$, the regression outcome is not determined by the data alone; it depends on the solution method. The slide presents this as central to understanding deep-learning success.

1. Stated in the context of overparameterized deep-learning regression.

2. The clip does not specify particular algorithms beyond the general claim.

1. Overparameterized deep learning as infinite conditioning

1. Caption evidence
   Observation

   Slide text states: "If we are overcomplete" / "linear regression is well defined".

2. Audio
   Observation

   Speaker says that with more data, spanning all dimensions, linear regression is well defined and can be solved by matrix inversion or gradient descent with the same solution.

If the data situation is overcomplete, then linear regression is well defined; moreover, solving by matrix inversion or by gradient descent yields the same solution.

1. The problem is in the linear-regression setting discussed on the slide.

2. The data situation is overcomplete.

3. The speaker additionally says there are more data and all dimensions are spanned.

Conditional statement about the overcomplete case.

1. Caption evidence
   Observation

   Slide text states: "If we are less than complete" / "it is undefined (and thus throws an error)".

2. Audio
   Observation

   Speaker says that with less data the answer is undefined and will throw an error depending on implementation, though most implementations will throw an error.

1. The exact error behavior is implementation-dependent according to the speaker.

If the data situation is less than complete, then the answer is undefined and thus throws an error in most implementations.

1. The problem is in the same linear-regression setting.

2. The data situation is less than complete / there is not enough data.

Conditional statement about the less-than-complete case.

1. Audio
   Observation

   Speaker says the mean squared error can still be made small, but there is no longer one solution; instead there are lots of solutions.

1. This is stated verbally without a displayed formula in the clip.

Even when the mean squared error can still be made small, the solution may fail to be unique; there can be lots of solutions.

1. Refers to the problematic case discussed immediately before the transition to ill-conditioned problems.

Existential statement about multiple solutions in the discussed setting.

1. Caption evidence
   Observation

   Slide title reads: "Ill conditioned problem: large or infinite condition number".

2. Audio
   Observation

   Speaker introduces condition number as quantifying how hard and how undefined a problem is.

1. The slide title asserts the association with large or infinite condition number, but the clip does not provide a formal proof.

An ill-conditioned problem is characterized on the slide by having a large or infinite condition number.

1. The problem is being considered in the condition-number framework introduced on the slide.

Defining/characterizing statement presented on the slide.

1. Audio
   Observation

   The lecturer says the condition number is the largest singular value of A divided by the smallest singular value.

2. Formula
   Observation

   Slide shows $κ(A) = σ$_max(A) / σ_min(A).

For the linear system Ax=b, the condition number of A is $κ(A)=σ$_max(A)/σ_min(A).

1. The problem is the linear-equation case shown on the slide.

2. Singular values of A are being used.

Stated for the matrix A in the displayed linear system.

1. Caption evidence
   Observation

   Slide states 'We usually have more parameters than data, so $κ = \infty$'.

2. Audio
   Observation

   The lecturer says in many cases we have more parameters than data.

1. The clip gives the conclusion but not a full derivation from parameter count to $κ=\infty$.

In many deep-learning settings, there are more parameters than data, and the slide summarizes this as $κ = \infty$.

1. The setting is deep-learning regression or inverse problems.

2. Parameter count exceeds data count.

Usually / in many cases, as stated by the slide and lecture.

1. Caption evidence
   Observation

   Slide states 'So regression generally depends on how we solve the problem!'

1. No explicit example of different solvers is shown within this clip.

In the overparameterized deep-learning regime, regression generally depends on how the problem is solved.

1. The problem is overparameterized.

2. The condition number is treated as infinite on the slide.

Generally, as written on the slide.

1. Audio
   Observation

   The lecturer says the condition number is infinity or very close to infinity because we are generally overparameterized.

2. Caption evidence
   Observation

   Slide 40 explicitly writes: "We usually have more parameters than data, so $κ = \infty$".

1. The precise matrix whose condition number is meant is not named in the visible slide text.

In the overparameterized linear DL setting shown here, having more parameters than data makes the condition number $\kappa$ equal to $\infty$.

1. The model has more parameters than data points.

2. The condition number is taken for the matrix associated with the linear regression/DL problem.

For the usual overparameterized cases discussed in the lecture.

1. Audio
   Observation

   The lecturer says regression generally depends on how we solve this problem.

2. Caption evidence
   Observation

   Slide 40 states: "So regression generally depends on how we solve the problem!"

In the overparameterized setting, the regression outcome generally depends on the optimization procedure used to solve the problem.

1. The regression problem is overparameterized.

2. There is not a unique solution determined by the data alone.

Generally, for the class of problems under discussion.

1. Audio
   Observation

   The lecturer says realizing there is a degree of freedom and not one solution is the key to understanding deep learning successes.

2. Caption evidence
   Observation

   Slide 40 states: "This is *the* key to understand DL successes".

Recognizing that overparameterized DL problems have a degree of freedom rather than a single solution is presented as the key to understanding deep learning successes.

1. The DL problem admits multiple solutions.

2. Success is being analyzed through the lens of solution freedom and optimization dynamics.

For the deep-learning successes discussed in this lecture segment.

1. Audio
   Observation

   The lecturer says the algorithm that we use for optimization matters and behavior may be non-trivial.

2. Caption evidence
   Observation

   Slide 41 states: "The algorithm used for optimization matters" and "Behavior may be non-trivial".

For linear DL, the optimization algorithm matters because the resulting behavior may be non-trivial.

1. One is studying linear DL dynamics.

2. The problem may have multiple solutions or non-simple optimization behavior.

In the linear DL settings considered in the lecture.

1. Formula
   Observation

   Slide shows $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$.

2. Audio
   Observation

   Speaker explains taking the limit as epsilon becomes small, looking at the largest value in that area of the ratio of the change in f, using the image of a ring around a middle value and asking for the steepest area within it.

1. The derivation is intuitive/geometric rather than a formal proof; the speaker does not derive the formula from earlier assumptions in this clip.

1. Expression

   $$$$
   Explanation

   Start from the displayed definition $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$.

   Justification

   Formula shown on the slide.

   Shown in the video
2. Expression

   $$$$
   Explanation

   Interpret $\varepsilon$ as the radius of a small neighborhood around the current value, which the speaker describes as a ring.

   Justification

   Speaker's verbal analogy while discussing the limit and the constraint $\|\delta x\| \le \varepsilon$.

   Shown in the video
3. Expression

   $$$$
   Explanation

   Inside that neighborhood, consider all allowed perturbations $\delta x$ and look at the ratio $\frac{\|\delta f\|}{\|\delta x\|}$.

   Justification

   Direct reading of the supremum expression on the slide.

   Shown in the video
4. Expression

   $$$$
   Explanation

   Take the largest such ratio within the neighborhood, matching the speaker's phrase about the steepest area.

   Justification

   Supremum in the displayed formula plus the speaker's informal description.

   Shown in the video
5. Expression

   $$$$
   Explanation

   Then let the neighborhood shrink by taking $\varepsilon \to 0$, which is the speaker's "as we make it small."

   Justification

   Limit in the displayed formula and corresponding audio explanation.

   Shown in the video

The lecturer's informal reading is that the condition number measures, in the infinitesimal limit, the worst-case amplification of a perturbation in $x$ into a perturbation in $f$. This is an interpretation of the displayed formula, not a separate proof.

1. Audio
   Observation

   The lecturer first describes changing x so that f changes most, then connects this to dimensions of high and low sensitivity, then states the singular-value formula.

2. Formula
   Observation

   The slide shows both the general κ formula and $κ(A)=σ$_max(A)/σ_min(A).

1. Expression

   $$$$
   Explanation

   Start from the question of how much the output f can change when the input x is perturbed slightly.

   Justification

   Audio: 'we could change x so that f changes most' and 'how small changes in the x ... produce changes in f'.

   Shown in the video
2. Expression

   $$\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$$
   Explanation

   Express that worst-case local sensitivity as a limiting supremum of output-to-input perturbation ratios.

   Justification

   Formula directly shown on the slide.

   Shown in the video
3. Expression

   $$$$
   Explanation

   Interpret the supremum geometrically: some directions in input space create large effects, others create small effects.

   Justification

   Audio explanation about dimensions where movement has a really big or really small effect.

   Shown in the video
4. Expression

   $$\kappa(A) = \frac{\sigma_{\max}(A)}{\sigma_{\min}(A)}$$
   Explanation

   Specialize to the linear system Ax=b, where the worst and best directional amplifications are encoded by the largest and smallest singular values.

   Justification

   Audio plus formula on slide.

   Shown in the video

The condition number measures worst-case local amplification, and for Ax=b it is represented by the ratio of largest to smallest singular value.

1. Audio
   Observation

   The lecturer moves from more parameters than data and $\kappa=\infty$ to overparameterization, then to regression depending on how the problem is solved, and finally to the importance of learning dynamics and degree of freedom.

2. Caption evidence
   Observation

   Slide 40 lists these statements in sequence: more parameters than data, $\kappa=\infty$, overparameterization, dependence on solver, and key to DL successes.

1. Expression

   $$\text{more parameters than data}$$
   Explanation

   Start from the regime in which the model contains more parameters than available data points.

   Justification

   Directly stated on slide 40 and spoken by the lecturer.

   Shown in the video
2. Expression

   $$\kappa = \infty$$
   Explanation

   The condition number of the associated matrix becomes infinite, indicating extreme ill-conditioning.

   Justification

   Explicit slide formula and audio statement that the condition number is infinity or very close to infinity.

   Shown in the video
3. Expression

   $$\text{generally overparameterized}$$
   Explanation

   This situation is identified as overparameterization.

   Justification

   Stated on slide 40 and repeated in the audio.

   Shown in the video
4. Expression

   $$\text{regression depends on how we solve the problem}$$
   Explanation

   Because the problem is overparameterized, the fitted regression is not fixed by the data alone; the chosen optimization procedure influences the solution reached.

   Justification

   Stated on slide 40 and explained verbally as the reason dynamics matter.

   Shown in the video
5. Expression

   $$\text{degree of freedom / not one solution}$$
   Explanation

   The lecturer interprets this dependence as a degree of freedom: there is not one solution, and recognizing this is central to understanding DL successes and regularization.

   Justification

   Spoken elaboration aligned with the final line of slide 40.

   Shown in the video

Overparameterization leads to $\kappa=\infty$, which makes the regression solution depend on the optimization dynamics; this freedom among solutions is presented as a key conceptual feature of deep learning.

1. Audio
   Observation

   The lecturer says this is why we should worry about dynamics of linear DL, that the optimization algorithm matters, and then asks about convergence and learning curves.

2. Caption evidence
   Observation

   Slide 41 displays the title about worrying about dynamics of linear DL, followed by algorithm matters, non-trivial behavior, and the questions about convergence and learning curves.

1. Expression

   $$\text{solution depends on optimization algorithm}$$
   Explanation

   Carry forward the conclusion that in overparameterized linear DL the algorithm affects which solution is obtained.

   Justification

   Established in the preceding slide and restated at the start of slide 41.

   Derived from the video
2. Expression

   $$\text{study dynamics of linear DL}$$
   Explanation

   Therefore the lecture turns to the dynamics of linear DL rather than only static regression formulas.

   Justification

   Explicit title and spoken transition on slide 41.

   Shown in the video
3. Expression

   $$\text{behavior may be non-trivial}$$
   Explanation

   The dynamics are not assumed to be simple; their behavior may be non-trivial.

   Justification

   Written on slide 41 and spoken by the lecturer.

   Shown in the video
4. Expression

   $$\text{ask about convergence and learning curves}$$
   Explanation

   The natural next questions are how convergence is affected and what learning curves look like under these dynamics.

   Justification

   Black-box prompt on slide 41 and corresponding spoken questions.

   Shown in the video

Because optimization algorithms affect the solution in overparameterized linear DL, the lecture motivates studying convergence and learning curves as non-trivial dynamical questions.

1. Diagram
   Observation

   Static slide titled "That is pretty interesting no?" with bullet text about overcomplete and less than complete cases; presenter webcam in top right; footer "Konrad Kording • Week 2" and page 38.

2. Animation
   Observation

   No slide-content animation occurs during this interval; only the presenter gestures in the webcam window.

1. Title text "That is pretty interesting no?"

2. Bullet text "If we are overcomplete"

3. Bullet text "linear regression is well defined"

4. Bullet text "If we are less than complete"

5. Bullet text "it is undefined (and thus throws an error)"

6. Presenter webcam overlay

7. Footer text "Konrad Kording • Week 2"

8. Page number 38

1. Presenter gestures with hands while speaking.

2. Slide content itself remains unchanged.

1. The four bullet lines remain visible throughout the interval.

2. The slide title and footer remain fixed.

The visual layout contrasts two regimes of linear regression: an overcomplete regime described as well defined, and a less-than-complete regime described as undefined and error-producing.

1. Diagram
   Observation

   At 50 seconds the slide changes to page 39 titled "Ill conditioned problem: large or infinite condition number" with formulas $Ax=b$, $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$, and $\kappa(A) = \frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$.

2. Animation
   Observation

   After the transition, the slide remains static while the presenter continues gesturing in the webcam window.

1. Title text "Ill conditioned problem: large or infinite condition number"

2. Formula $Ax=b$

3. Formula $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$

4. Formula $\kappa(A) = \frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$

5. Presenter webcam overlay

6. Footer text "Konrad Kording • Week 2"

7. Page number 39

1. Slide content switches from page 38 to page 39 at 50 seconds.

2. Presenter raises hands and makes circular/ring-like gestures while explaining the limit and neighborhood idea.

1. Once page 39 appears, its three displayed formulas remain on screen through the end of the clip.

2. The title and footer remain fixed after the transition.

The second slide shifts from qualitative contrast of data regimes to a formal condition-number framework for ill-conditioned linear systems.

1. Diagram
   Observation

   Static slide 39 with title 'Ill conditioned problem: large or infinite condition number' and three displayed formulas.

2. Animation
   Observation

   Only the lecturer's webcam gestures change; the slide content remains static.

1. Title text

2. Equation Ax=b

3. General κ formula

4. Matrix $κ(A)$ formula

5. Footer 'Konrad Kording • Week 2'

6. Slide number 39

7. Speaker webcam inset

1. Speaker gestures with hands while explaining.

2. No mathematical content changes on the slide during this interval.

1. All three formulas remain visible throughout the interval.

2. The slide title remains unchanged until 88 seconds.

The visual anchors the lecture on two levels of definition: a general perturbation-based condition number and its matrix specialization through singular values.

1. Diagram
   Observation

   Slide changes to page 40 titled 'DL is weird here' with four bullet-like statements.

2. Animation
   Observation

   The transition occurs at about 88 seconds; after that the new slide stays static.

1. Title 'DL is weird here'

2. Statement 'We usually have more parameters than data, so $κ = \infty$'

3. Statement 'We are generally overparameterized'

4. Statement 'So regression generally depends on how we solve the problem!'

5. Statement 'This is *the* key to understand DL successes'

6. Footer 'Konrad Kording • Week 2'

7. Slide number 40

8. Speaker webcam inset

1. At 88 seconds the presentation switches from slide 39 to slide 40.

2. Speaker continues gesturing while discussing parameter counts and data dimensionality.

1. The four textual claims remain on screen after the transition.

2. The footer and slide number remain fixed on slide 40.

The second slide applies the conditioning discussion to deep learning, asserting overparameterization, infinite κ, and solver-dependent regression outcomes.

1. Diagram
   Observation

   A static lecture slide titled "DL is weird here" shows four bullet lines including "We usually have more parameters than data, so $κ = \infty$", "We are generally overparameterized", "So regression generally depends on how we solve the problem!", and "This is *the* key to understand DL successes".

2. Animation
   Observation

   The lecturer appears in a small webcam box at the upper right and gestures while speaking; the slide content itself remains static during this interval.

1. Title "DL is weird here"

2. Bullet text about more parameters than data

3. Formula $\kappa = \infty$

4. Bullet text about overparameterization

5. Bullet text about regression depending on solver

6. Bullet text about key to DL successes

7. Webcam view of lecturer

1. Lecturer gestures change over time while the slide text remains fixed.

1. Slide title and all four bullet lines remain visible throughout 0–40 s.

2. The formula shown is $\kappa = \infty$.

The visual layout emphasizes a logical chain from parameter excess to infinite condition number, then to solver-dependent regression and the conceptual importance of solution freedom.

1. Diagram
   Observation

   At 40 s the slide changes to one titled "This is why we should worry about dynamics of linear DL" with bullets "The algorithm used for optimization matters" and "Behavior may be non-trivial", plus a black box reading "So what about convergence? Learning curves?".

2. Animation
   Observation

   The new slide remains static after the transition; the lecturer continues speaking in the webcam box.

1. Title "This is why we should worry about dynamics of linear DL"

2. Bullet "The algorithm used for optimization matters"

3. Bullet "Behavior may be non-trivial"

4. Black box with "So what about convergence? Learning curves?"

5. Webcam view of lecturer

1. At approximately 40 s the presentation switches from slide 40 to slide 41.

1. After the switch, slide 41 text remains unchanged until the end of the clip.

2. The black box visually highlights the convergence and learning-curve questions.

The second slide converts the previous conceptual point into a research-style question: if the optimizer matters, then convergence and learning curves must be analyzed dynamically.

1. Audio
   Observation

   Speaker says the answer is undefined and will actually throw us an error, depending on the implementation, but in most implementations it will throw us an error.

One might assume that an undefined linear-regression problem always produces the same explicit error in every software system.

The lecturer explicitly says the behavior depends on implementation, although most implementations will throw an error.

1. Audio
   Observation

   Speaker says the mean squared error can still be made small, but there is no longer one solution; there are lots of solutions.

One might infer that if the mean squared error can be made small, the solution must be unique.

The lecturer states the opposite: the objective value can still become small while uniqueness is lost and many solutions exist.

1. Audio
   Observation

   The lecturer says our intuitions are really good in low dimensions and really bad in high dimensions, and that condition numbers often surprise us.

One may assume that geometric intuition from low-dimensional spaces reliably predicts sensitivity and conditioning in high-dimensional problems.

The lecturer explicitly warns that intuition is much less reliable in high dimensions, so condition-number behavior can be surprising.

1. Caption evidence
   Observation

   Slide states 'So regression generally depends on how we solve the problem!'

One may think the fitted regression is fixed solely by the data once a model class is chosen.

In the overparameterized deep-learning regime described on slide 40, the solution generally depends on the method used to solve the problem.

1. Audio
   Observation

   The lecturer says regression generally depends on how we solve the problem and that there is not one solution, but a degree of freedom.

2. Caption evidence
   Observation

   Slide 40 states that regression generally depends on how we solve the problem and calls this the key to understanding DL successes.

One might think that once the data and model class are fixed, regression has a single solution independent of the algorithm.

In the overparameterized case with $\kappa=\infty$, the lecture states that there is not one solution; the result depends on the optimization dynamics and therefore on the solver used.

1. Audio
   Observation

   The lecturer says the algorithm used for optimization matters and behavior may be non-trivial.

2. Caption evidence
   Observation

   Slide 41 explicitly states: "The algorithm used for optimization matters" and "Behavior may be non-trivial".

One might treat the optimization algorithm as a mere implementation detail that does not affect the learned model.

The lecture argues the opposite for linear DL in this setting: the optimizer shapes the dynamics, the behavior can be non-trivial, and convergence and learning curves become central questions.

1. Caption evidence
   Observation

   Slide 38 lists the overcomplete case and the less-than-complete case as contrasting bullets.

2. Audio
   Observation

   Speaker first explains the overcomplete case and then contrasts it with having less data.

The lecture presents the overcomplete case and the less-than-complete case as opposite regimes for whether linear regression is well defined or undefined.

1. Audio
   Observation

   After discussing many solutions, the speaker says there is a way of thinking a little more precisely about such problems and introduces ill-conditioned problems and the condition number.

2. Diagram
   Observation

   Slide changes to "Ill conditioned problem: large or infinite condition number".

The condition-number framework is introduced as a more precise way to analyze the kind of non-uniqueness / indefiniteness just discussed.

1. Formula
   Observation

   Both $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$ and $\kappa(A) = \frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$ appear together on the same slide under the heading about ill-conditioned problems.

1. The clip shows both formulas together but does not verbally derive one from the other.

Within the displayed linear-system context, the slide presents the general limiting-supremum definition and the matrix-specific singular-value ratio as two expressions of the condition number.

1. Formula
   Observation

   Slide shows the general κ formula above the specialized $κ(A)$ formula.

2. Audio
   Observation

   The lecturer moves from general sensitivity language to the linear-equation case.

The matrix formula $κ(A)=σ$_max(A)/σ_min(A) is presented as the linear-system specialization of the broader perturbation-based condition number idea.

1. Caption evidence
   Observation

   Slide title links 'Ill conditioned problem' with 'large or infinite condition number'.

The notion of an ill-conditioned problem is defined through having a large or infinite condition number.

1. Audio
   Observation

   The lecturer says the notion applies broadly to inverse problems and in this case to solving deep-learning problems.

2. Caption evidence
   Observation

   Slide 40 applies $κ = \infty$ to overparameterized deep learning.

The lecture transfers the conditioning framework from linear inverse problems to deep-learning regression, where overparameterization is summarized as $κ = \infty$.

1. Caption evidence
   Observation

   Slide 40 lists overparameterization and then states that regression depends on how the problem is solved.

The slide presents solver-dependence of regression as a consequence of the overparameterized, infinite-condition-number regime.

1. Caption evidence
   Observation

   Slide 40 writes "We usually have more parameters than data, so $κ = \infty$".

2. Audio
   Observation

   The lecturer links overparameterization directly to the condition number being infinity or very close to infinity.

The proposition that $\kappa=\infty$ is presented as a direct consequence of having more parameters than data in the overparameterized linear DL setting.

1. Audio
   Observation

   The lecturer moves from bad/infinite condition number and overparameterization to the statement that regression depends on how the problem is solved.

2. Caption evidence
   Observation

   Slide 40 orders these ideas as consecutive bullets.

The infinite condition number / overparameterized regime is the reason the lecture gives for why the regression solution depends on the solver.

1. Audio
   Observation

   The lecturer explains that because there is not one solution, there is a degree of freedom, and this is the key to understanding DL successes.

2. Caption evidence
   Observation

   Slide 40 ends with "This is *the* key to understand DL successes" immediately after the solver-dependence bullet.

The idea of a degree of freedom is contained in, and motivated by, the earlier claim that regression depends on the optimization method.

1. Audio
   Observation

   The lecturer says this is why we should worry about the dynamics of linear DL after discussing degree of freedom and multiple solutions.

2. Caption evidence
   Observation

   Slide 41 title states "This is why we should worry about dynamics of linear DL".

Recognizing multiple solutions and solver dependence is the prerequisite motivation for studying the dynamics of linear DL.

1. Audio
   Observation

   The lecturer asks how this affects convergence and how learning curves look in these cases.

2. Caption evidence
   Observation

   Slide 41 black box reads "So what about convergence? Learning curves?".

Once dynamics are deemed important, the lecture applies that viewpoint to the concrete questions of convergence and learning-curve behavior.

1. Audio
   Observation

   Speaker says the condition number quantifies how hard and how undefined a problem is.

2. Caption evidence
   Observation

   Slide title mentions large or infinite condition number.

1. Condition number quantifies hardness and indefiniteness

1. Formula
   Observation

   Slide displays $\kappa = \lim_{\varepsilon \to 0} \sup_{\|\delta x\| \le \varepsilon} \frac{\|\delta f\|}{\|\delta x\|}$.

1. Supremum-limit definition of the condition number

1. Formula
   Observation

   Slide displays $\kappa(A) = \frac{\sigma_{\max(A)}}{\sigma_{\min(A)}}$.

1. Condition number of A as ratio of extreme singular values

1. Caption evidence
   Observation

   Slide 38 contrasts "If we are overcomplete" with "If we are less than complete".

2. Audio
   Observation

   Speaker explains one case as well defined and the other as undefined/error-producing.

1. Overcomplete case makes linear regression well defined
2. Less-than-complete case makes the answer undefined

1. Audio
   Observation

   Speaker says the mean squared error can still be made small but there are lots of solutions.

1. Mean squared error can still be made small despite non-uniqueness

1. Audio
   Observation

   Speaker describes a ring around a middle value and asks what the steepest area within that is.

2. Formula
   Observation

   The corresponding displayed formula contains $\lim_{\varepsilon \to 0}$ and $\sup_{\|\delta x\| \le \varepsilon}$.

1. Supremum-limit definition of the condition number
2. Informal interpretation of the supremum-limit formula

1. Formula
   Observation

   General κ formula is displayed on slide 39.

2. Audio
   Observation

   Lecturer explains it as the effect of small changes in x on f.

1. General condition number formula
2. Ill-conditioned problem

1. Formula
   Observation

   Slide shows $κ(A)=σ$_max(A)/σ_min(A).

2. Audio
   Observation

   Lecturer states the same ratio verbally.

1. Condition number of a matrix via singular values

1. Audio
   Observation

   Lecturer says intuition is good in low dimensions and bad in high dimensions.

1. Low-dimensional intuition fails in high dimensions
2. Directional intuition for conditioning

1. Caption evidence
   Observation

   Slide 40 states more parameters than data implies $κ = \infty$.

1. Overparameterized deep learning as infinite conditioning

1. Caption evidence
   Observation

   Slide 40 states regression generally depends on how we solve the problem.

1. Solution dependence in overparameterized regression

1. Caption evidence
   Observation

   Slide 40 states "We usually have more parameters than data, so $κ = \infty$".

2. Audio
   Observation

   The lecturer says the condition number is infinity or very close to infinity in overparameterized cases.

1. Overparameterization makes the condition number infinite
2. More parameters than data implies $\kappa=\infty$