Uncertainty Propagation: Linear Tangent

1. Diagram
   Observation

   The vertical axis of the graph is labeled Y.

Y

Output variable or dependent variable in the nonlinear model.

Real numbers

1. Diagram
   Observation

   The horizontal axis of the graph is labeled $\Theta$.

$\Theta$

Input variable or parameter with uncertainty.

Real numbers

1. Formula
   Observation

   The equation $y = f(x|\Theta)$ is displayed on the graph.

f(x|$\Theta$)

Nonlinear function relating output y to input x and parameter $\Theta$.

Real-valued function

1. Formula
   Observation

   The expression $Var[y] \sim (dy/d \Theta)^2 Var[\Theta]$ is shown.

Var[y]

Variance of the output variable y.

Non-negative real numbers

1. Formula
   Observation

   The derivative term $(dy/d \Theta)$ appears in the variance approximation formula.

$dy/d\Theta$

Derivative of y with respect to $\Theta$, representing the slope of the linear tangent.

Real numbers

1. Formula
   Observation

   The term $Var[\Theta]$ appears in the variance approximation formula.

Var[$\Theta$]

Variance of the input parameter $\Theta$.

Non-negative real numbers

1. Formula
   Observation

   The Taylor series expansion $Var[f(x|\Theta)] \approx Var\left[f(x|\bar{\Theta}) + \frac{df}{d\Theta}(x|\bar{\Theta})(\Theta - \bar{\Theta}) + \dots\right]$ is displayed.

Var[f(x|$\Theta$)]

Variance of the function $f(x|\Theta)$ approximated using a Taylor series expansion.

Non-negative real numbers

1. Formula
   Observation

   Var[f(x|θ)]

Var[f(x|θ)]

Variance of the model output f given input x and parameter θ

Statistical variance

1. Formula
   Observation

   f(x|θ̄) + (df/dθ)(x|θ̄)/1! (θ - θ̄) + ...

f(x|θ̄) + $\frac{\frac{df}{d\theta}(x|\bar{\theta})}{1!}(\theta - \bar{\theta})$ + ...

First-order Taylor series expansion of the function f around the mean parameter value θ̄

Calculus

1. Formula
   Observation

   (∂f/∂$θ_i$)^2

$\left(\frac{\partial f}{\partial \theta_i}\right)^2$

Square of the partial derivative of the function f with respect to the i-th parameter $θ_i$

Calculus

1. Formula
   Observation

   var[$θ_i$]

var[$\theta_i$]

Variance of the i-th parameter $θ_i$

Statistics

1. Formula
   Observation

   cov[$θ_i$, $θ_j$]

cov[$\theta_i$, $\theta_j$]

Covariance between the i-th and j-th parameters

Statistics

1. Audio
   Observation

   The speaker explains that for nonlinear models, uncertainty can be propagated by taking the linear tangent of the model.

2. Diagram
   Observation

   A graph shows a nonlinear curve with a linear tangent line drawn at a specific point.

To propagate uncertainty through a nonlinear model, the model is approximated by its linear tangent at the mean value of the input parameter. This allows the use of rules for linear models to estimate the variance of the output.

1. The input parameter $\Theta$ has a known mean and variance.

2. The function $f(x|\Theta)$ is differentiable with respect to $\Theta$.

1. Audio
   Observation

   The speaker mentions replacing the function with a Taylor series expansion focusing on the first few terms.

2. Formula
   Observation

   The Taylor series expansion formula is displayed on the screen.

The nonlinear function $f(x|\Theta)$ is expanded using a Taylor series around the mean value $\bar{\Theta}$. The first few terms of this expansion are used to approximate the function as a linear model.

1. The function $f(x|\Theta)$ is infinitely differentiable with respect to $\Theta$.

2. The expansion is centered at the mean value $\bar{\Theta}$.

1. Linear Tangent Approximation

1. Formula
   Observation

   Var[f(x|θ)] ≈ Var[f(x|θ̄) + (df/dθ)(x|θ̄)/1! (θ - θ̄) + ...]

2. Audio
   Observation

   The speaker explains that in a nonlinear model, we use derivatives instead of specific slopes, and these derivatives may change depending on where you are in parameter space.

To propagate uncertainty through a nonlinear model, the model function is approximated using a first-order Taylor series expansion around the mean parameter values. This linearizes the function locally, allowing the use of standard variance propagation rules.

1. The model must be differentiable with respect to its parameters.

2. The expansion is valid locally around the mean parameter values θ̄.

1. Formula
   Observation

   var[$f(x)$] ≈ (∂f/∂$θ_i$)^2 var[$θ_i$]

2. Audio
   Observation

   Each source of uncertainty... is represented in terms of its direct component where there's the variance of that term multiplied by the square of the derivative of how our output responds to our input.

The contribution of a single parameter's uncertainty to the total output variance is calculated as the product of the parameter's variance and the square of the partial derivative of the model output with respect to that parameter. This represents the sensitivity of the output to that specific input.

1. Assumes first-order Taylor approximation.

2. Applies to independent parameters or isolates the direct effect of one parameter.

1. First-Order Taylor Expansion for Nonlinear Models

1. Formula
   Observation

   var[$f(x)$] ≈ Σ (∂f/∂$θ_i$)^2 var[$θ_i$] + $Σ_{i\ne j}$ (∂f/∂$θ_i$)(∂f/∂$θ_j$) cov[$θ_i$, $θ_j$]

2. Audio
   Observation

   We also need to include terms that account for the interactions that arise as the product of the partial derivatives of each of two factors and then the covariance between those two factors.

When parameters are correlated, their joint uncertainty contributes to the output variance. This interaction effect is captured by summing over all pairs of distinct parameters, multiplying their respective partial derivatives by their covariance.

1. Parameters may be correlated.

2. Requires calculation of pairwise covariances and partial derivatives.

1. Direct Variance Component in Uncertainty Propagation

1. Audio
   Observation

   The speaker describes the process of approximating a nonlinear model with its linear tangent to propagate uncertainty.

2. Diagram
   Observation

   The graph visually demonstrates the concept of a linear tangent to a nonlinear curve.

1. Expression

   $$$$
   Explanation

   Identify the nonlinear function $f(x|\Theta)$ and the mean value $\bar{\Theta}$ of the input parameter.

   Justification

   The goal is to approximate the function locally around the mean value.

   Shown in the video
2. Expression

   $$$$
   Explanation

   Compute the derivative of the function with respect to $\Theta$ at the mean value, $(dy/d \Theta)|_{\Theta=\bar{\Theta}}$.

   Justification

   The derivative represents the slope of the linear tangent at the mean value.

   Shown in the video
3. Expression

   $$$$
   Explanation

   Use the linear tangent approximation to estimate the variance of the output $y$: $Var[y] \sim (dy/d \Theta)^2 Var[\Theta]$.

   Justification

   For a linear function, the variance of the output is the square of the slope times the variance of the input.

   Shown in the video

The variance of the output of a nonlinear function can be approximated by using the linear tangent at the mean value of the input parameter.

1. Formula
   Observation

   Sequence of formulas showing Taylor expansion leading to variance decomposition.

2. Audio
   Observation

   Speaker narrates the transition from linear models to nonlinear models using derivatives and explains the components of the final variance equation.

1. Expression

   $$Var[f(x|\theta)] \approx Var\left[f(x|\bar{\theta}) + \frac{\frac{df}{d\theta}(x|\bar{\theta})}{1!}(\theta - \bar{\theta}) + \dots\right]$$
   Explanation

   Start with the nonlinear function f(x|θ) and apply a first-order Taylor expansion around the mean parameter vector θ̄ to linearize it locally.

   Justification

   Definition of Taylor series expansion.

   Shown in the video
2. Expression

   $$var[f(x)] \approx \left(\frac{\partial f}{\partial \theta_i}\right)^2 var[\theta_i]$$
   Explanation

   Identify the direct contribution of the i-th parameter's variance to the output variance, scaled by the square of its sensitivity (partial derivative).

   Justification

   Standard rule for propagating variance through a linear transformation.

   Shown in the video
3. Expression

   $$var[f(x)] \approx \sum \left(\frac{\partial f}{\partial \theta_i}\right)^2 var[\theta_i] + \sum_{i \neq j} \left(\frac{\partial f}{\partial \theta_i}\right) \left(\frac{\partial f}{\partial \theta_j}\right) cov[\theta_i, \theta_j]$$
   Explanation

   Sum the direct variance contributions for all parameters and add the interaction terms arising from parameter correlations, represented by covariances.

   Justification

   Properties of variance for sums of random variables, including covariance terms for dependent variables.

   Shown in the video

The approximate variance of a nonlinear model's output can be decomposed into a sum of individual parameter variance contributions weighted by squared sensitivities, plus interaction terms weighted by parameter covariances.

1. Diagram
   Observation

   A graph shows a nonlinear curve with a linear tangent line drawn at a specific point. The axes are labeled Y and $\Theta$.

1. Nonlinear curve

2. Linear tangent line

3. Axes labeled Y and $\Theta$

1. The linear tangent line is drawn at a specific point on the nonlinear curve.

1. The nonlinear curve remains unchanged.

2. The axes labels remain unchanged.

The graph illustrates how a nonlinear function can be locally approximated by a linear tangent line to estimate the propagation of uncertainty.

1. Formula
   Observation

   The Taylor series expansion formula is displayed on the screen.

1. Taylor series expansion formula

1. The formula appears on the screen.

1. No other visual elements change.

The formula provides the mathematical basis for approximating a nonlinear function using a series expansion around a mean value.

1. Diagram
   Observation

   Static display of the Taylor expansion formula inside a variance operator.

1. Mathematical formula

1. The formula remains static on screen while the speaker introduces the concept.

Visually anchors the mathematical definition of linearizing a nonlinear model using a Taylor series.

1. Diagram
   Observation

   Static display of the single-parameter variance propagation formula.

1. Mathematical formula

1. The formula remains static on screen during explanation.

Highlights the core mechanism of how a single parameter's uncertainty scales by model sensitivity.

1. Diagram
   Observation

   Static display of the full variance propagation formula including summations and covariance terms.

1. Mathematical formula

1. The formula remains static on screen during explanation.

Presents the complete analytical expression for approximating output variance, combining direct and interaction effects.

1. Audio
   Observation

   The key difference is in our linear model we had specific slopes and in we do a nonlinear model we have instead the derivatives and those derivatives may change depending on where you are in parameter space.

Learners might assume that nonlinear models have constant slopes like linear models.

In nonlinear models, the 'slope' is replaced by derivatives which vary depending on the location in the parameter space.

1. Audio
   Observation

   The speaker connects the linear tangent approximation to the Taylor series expansion.

The linear tangent approximation is a specific application of the first-order term in the Taylor series expansion of a nonlinear function.

1. Audio
   Observation

   Speaker transitions from discussing the Taylor expansion to explaining the direct variance component.

The first-order Taylor expansion provides the linear approximation necessary to apply standard variance propagation rules to individual parameters.

1. Audio
   Observation

   Speaker says 'And as before, we also need to include terms that account for the interactions...' after explaining the direct component.

The full variance propagation formula contains the direct variance components as its first summation term, augmented by the covariance interaction terms.

1. Audio
   Observation

   The speaker discusses the linear tangent approximation for propagating uncertainty in nonlinear models.

1. Linear Tangent Approximation

1. Audio
   Observation

   The speaker mentions the Taylor series expansion as a method to approximate nonlinear functions.

1. Taylor Series Expansion

1. Audio
   Observation

   Explanation of using derivatives instead of slopes for nonlinear models.

1. First-Order Taylor Expansion for Nonlinear Models

1. Formula
   Observation

   Formula showing the covariance summation term.

1. Interaction Terms via Covariance in Uncertainty Propagation