K-Fold Cross Validation - Intro to Machine Learning

1. Audio
   Observation

   The speaker says the data set is partitioned into k bins of equal size.

2. Formula
   Observation

   The board shows a red lowercase k beside the segmented bar and later writes $k = 10$.

k

Number of equal-size folds or bins used in k-fold cross-validation.

Integer number of partitions; in the worked example $k = 10$.

1. Audio
   Observation

   The speaker gives an example with 200 data points.

2. Formula
   Observation

   The board writes 200 = data set size.

200 = data set size

Total number of data points in the example dataset.

Positive integer count of observations.

1. Audio
   Observation

   The speaker says, if you have 10 bins, what is the number of data points per bin?

2. Formula
   Observation

   The board writes $10 = k$.

$10 = k$

In the example, the number of folds is 10.

Specific instance of k.

1. Audio
   Observation

   The speaker answers that each bin has 20 data points.

2. Formula
   Observation

   The board writes 20 inside a small box and then labels the visible bins with 20.

20

Number of data points per fold in the example with 200 total points and 10 folds.

Derived from $200 / 10$.

1. Formula
   Observation

   The board shows the text '$k = 10$' circled in red.

2. Audio
   Observation

   The speaker mentions '10-fold cross validation'.

k

The number of folds used in k-fold cross validation.

Positive integer

1. Formula
   Observation

   The speaker writes '10-fold C.V.' on the whiteboard.

C.V.

Abbreviation for Cross Validation.

Text abbreviation

1. Audio
   Observation

   The speaker introduces a method called cross validation as something that slightly generalizes training and test sets.

2. Formula
   Observation

   The title CROSS VALIDATION is written at the top of the board.

Cross validation is presented as a generalization of the simple training/test split idea for evaluating machine learning models.

1. Used when one wants to evaluate model performance while addressing the tradeoff between training-set size and testing-set size.

1. Audio
   Observation

   The speaker discusses problems with splitting a data set into training and testing data and describes the dilemma of maximizing both sets.

2. Formula
   Observation

   The board writes Problems with splitting into training & testing data, then draws a long bar divided into test and training portions with arrows labeled max.

A single split forces a choice of what fraction of the data is used for testing and what fraction for training. The speaker frames this as a dilemma because one wants many points in the training set for the best learning result and also many points in the test set for the best validation.

1. Applies to a fixed partition of the dataset into a training subset and a testing subset.

1. Audio
   Observation

   The speaker states there is an inherent trade-off: every data point taken out of the training set into the test set is lost for the training set.

2. Formula
   Observation

   The board circles the boundary between the test and training portions of the bar.

The video identifies an inherent trade-off in a single split: moving any observation from training to testing reduces the amount of data available for training.

1. Assumes the total dataset size is fixed.

1. Problem with a single train-test split

1. Audio
   Observation

   The speaker says the basic idea is to partition the data set into k bins of equal size and run k separate learning experiments.

2. Formula
   Observation

   The board writes k-fold CROSS VALIDATION, shows $k = 10$, and lists Run k seperate learning experiments, pick testing set, train, test on testing set, Average test results from those k experiments.

k-fold cross validation partitions the dataset into k equal-size bins, then repeatedly uses one bin as the testing set and the remaining k-1 bins as the training set, runs the learning algorithm, evaluates on the held-out bin, and averages the k test results.

1. The dataset is partitioned into k bins of equal size.

2. Each experiment uses one bin for testing and the remaining k-1 bins for training.

3. The final reported result is the average over the k experiments.

1. Cross validation
2. Problem with a single train-test split

1. Audio
   Observation

   The speaker gives the example of 200 data points and 10 bins and asks how many data points are in each bin.

2. Formula
   Observation

   The board writes 200 = data set size, $10 = k$, and 20 in a box, then labels the bins with 20.

For a dataset of 200 points divided into 10 equal-size bins, each bin contains 20 data points.

1. Total dataset size is 200.

2. Number of bins is 10.

3. Bins are equal size.

1. k-fold cross validation

1. Audio
   Observation

   The speaker says the key thing in cross validation is to run this multiple times and then average the different testing set performances.

2. Formula
   Observation

   The board writes Average test results from those k experiments.

After running the k separate experiments, the performance measured on each held-out testing set is averaged to produce the overall cross-validation estimate.

1. There are k separate test results, one from each fold.

1. k-fold cross validation

1. Formula
   Observation

   The board lists the steps: 'Run k separate learning experiments', '- pick testing set', '- train', '- test on testing set', and 'Average test results from those k experiments'.

2. Audio
   Observation

   The speaker explains that you run k separate learning experiments and average the test results.

The method involves running k separate learning experiments. For each experiment, a testing set is picked, the model is trained, and then tested on that specific testing set. Finally, the test results from all k experiments are averaged.

1. The dataset must be divided into k subsets or folds.

1. Formula
   Observation

   The speaker writes 'min. training time', 'min. run time', and 'max accuracy' on the left side of the board.

2. Audio
   Observation

   The speaker lists minimizing training time, minimizing run time after training, and maximizing accuracy as goals.

When assessing machine learning algorithms, common goals include minimizing the time it takes to train the model, minimizing the runtime required to make predictions after training, and maximizing the accuracy of the model's predictions.

1. Audio
   Observation

   The speaker says there is an inherent trade-off because every data point taken out of the training set into the test set is lost for the training set.

With a fixed dataset, increasing the test set by moving points out of the training set necessarily decreases the training set size.

1. The dataset is split into training and testing subsets.

2. The total number of data points is fixed.

For every data point moved from training to testing.

1. Audio
   Observation

   The speaker explains that in k-fold cross validation one partitions the data into k equal-size bins, runs k separate learning experiments, uses one bin as testing and the remaining k-1 bins as training, then averages the k test results.

2. Formula
   Observation

   The board lists the procedure: Run k seperate learning experiments; pick testing set; train; test on testing set; Average test results from those k experiments.

k-fold cross validation consists of partitioning the data into k equal-size bins, then for each of k experiments using one bin as the testing set and the remaining k-1 bins as the training set, training and testing, and finally averaging the k test results.

1. The dataset can be partitioned into k bins of equal size.

2. Each experiment uses exactly one held-out bin for testing.

For each of the k bins, that bin is used once as the testing set.

1. Audio
   Observation

   The speaker states that the assessment of your learning algorithm will be more accurate.

Using k-fold cross validation provides a more accurate assessment of a learning algorithm compared to a static train/test split.

1. The algorithm is evaluated using k separate learning experiments.

For any given dataset and learning algorithm.

1. Audio
   Observation

   The speaker says, 'you've kind of used all your data for training and all your data for testing'.

In k-fold cross validation, all available data is utilized for both training and testing across the different folds.

1. The dataset is partitioned into k folds.

Across all k experiments.

1. Audio
   Observation

   The speaker asks, if you have 200 data points and 10 bins, what is the number of data points per bin? and answers 20.

2. Formula
   Observation

   The board shows 200 = data set size, $10 = k$, and 20 in a box.

1. Expression

   $$200 = data set size$$
   Explanation

   The example starts with a dataset containing 200 data points.

   Justification

   Stated directly in the audio and written on the board.

   Shown in the video
2. Expression

   $$10 = k$$
   Explanation

   The example uses 10 bins.

   Justification

   Stated directly in the audio and written on the board.

   Shown in the video
3. Expression

   $$20$$
   Explanation

   Each bin contains 20 data points.

   Justification

   The speaker states the answer verbally and writes 20 on the board; the underlying arithmetic is 200 divided by 10.

   Shown in the video

For 200 data points partitioned into 10 equal-size bins, each bin has 20 data points.

1. Audio
   Observation

   The speaker gives an example with 200 data points and 10 bins and says each bin has 20 data points.

2. Formula
   Observation

   The board writes 200 = data set size, $10 = k$, and 20 in a box, then labels the bins with 20.

Given a dataset of 200 data points partitioned into 10 equal-size bins, determine the number of data points per bin.

1. data set size = 200

2. $k = 10$

3. bins are equal size

Find the number of data points in each bin.

1. Expression

   $$200 = data set size$$
   Explanation

   Use the stated total dataset size.

   Justification

   Given in the example.

   Shown in the video
2. Expression

   $$10 = k$$
   Explanation

   Use the stated number of bins.

   Justification

   Given in the example.

   Shown in the video
3. Expression

   $$20$$
   Explanation

   Each bin contains 20 data points.

   Justification

   Computed by dividing the total number of points equally among the 10 bins.

   Shown in the video

20 data points per bin.

The board explicitly writes 20 and labels the bins with 20.

1. Animation
   Observation

   A hand writes the blue title CROSS VALIDATION and underlines it, then writes the black heading Problems with splitting into training & testing data.

1. Blue title text CROSS VALIDATION

2. Black heading Problems with splitting into training & testing data

1. The title is written first.

2. The problem heading is added below the title.

1. The board remains a white background with handwritten text.

The visual sequence establishes the topic and motivates the discussion by naming the issue with a single train-test split.

1. Animation
   Observation

   A long red horizontal bar is drawn, the left portion is shaded and labeled test, the right portion is dotted and labeled training, and green arrows labeled max point downward from both regions.

2. Animation
   Observation

   The speaker’s pen circles the boundary between the test and training portions.

1. Long red bar representing the dataset

2. Shaded left segment labeled test

3. Dotted right segment labeled training

4. Green arrows labeled max

5. Green circle around the split boundary

1. The dataset bar is drawn.

2. The test and training portions are distinguished visually.

3. Arrows indicate the desire to maximize both portions.

4. The boundary is circled to emphasize the tradeoff.

1. The bar represents one fixed dataset.

2. The two portions together make up the whole bar.

The diagram shows that in a single split, enlarging one subset comes at the expense of the other because the total dataset length is fixed.

1. Animation
   Observation

   The board is cleared except for the title, then a new red bar is divided into equal segments and labeled with k.

2. Animation
   Observation

   The example values 200 = data set size, $10 = k$, and 20 are written, followed by a two-row grid of five visible boxes each labeled 20.

3. Animation
   Observation

   The top-left box is shaded green, $k = 10$ is written, and the procedure list is added below.

1. Only five bins are visibly drawn in each row although the spoken and written example uses $k = 10$.

1. Red segmented bar

2. Red lowercase k

3. Text 200 = data set size

4. Text $10 = k$

5. Boxed 20

6. Two-row grid of visible bins labeled 20

7. Green-shaded top-left bin

8. Procedure list in green and black

1. The single-split diagram is replaced by a partitioned bar.

2. The example numbers are written.

3. A grid of equal-size bins is drawn.

4. One bin is highlighted as the testing set.

5. The algorithm steps are listed beneath the grid.

1. All visible bins are labeled with the same size 20.

2. The highlighted bin is singled out as the testing set while the others remain available for training.

The visual model conveys equal partitioning into folds and the repeated holdout pattern used in k-fold cross-validation.

1. Animation
   Observation

   The whiteboard content abruptly changes from the k-fold diagram to a blank screen with only the title 'CROSS VALIDATION'.

1. Whiteboard content

1. The previous diagram and text disappear.

1. The title 'CROSS VALIDATION' remains at the top.

The instructor clears the board to start a new comparison exercise.

1. Animation
   Observation

   The speaker draws small circles under the 'TRAIN/TEST' and '10-fold C.V.' columns, aligning them with the three goals listed on the left.

1. Circles

2. Text labels

1. Six circles are drawn to form a grid for selecting the best method for each goal.

1. The text labels for the methods and goals remain unchanged.

A visual matrix is created to map the evaluation goals to the two different validation methodologies.

1. Audio
   Observation

   The speaker says you would like to maximize both sets, but there is an inherent trade-off because every point moved into the test set is lost for training.

2. Formula
   Observation

   The board places max arrows under both test and training and circles the boundary between them.

One may want both the training set and the testing set to be as large as possible at the same time.

With a fixed dataset, a single split cannot maximize both subsets simultaneously; increasing one reduces the other.

1. Audio
   Observation

   The speaker says, So obviously this takes more compute time because you now have to run...

1. The sentence is cut off before the full consequence is stated.

Cross-validation might be treated as cost-free compared with a single train-test split.

The video explicitly notes that this approach takes more compute time because the learning experiment is run multiple times.

1. Audio
   Observation

   The speaker says cross validation slightly generalizes the earlier training and test set idea, then explains the problem with a single split.

Cross validation is introduced as a generalization of the simpler train-test split framework.

1. Audio
   Observation

   The speaker moves from the general term cross validation to the specific k-fold procedure.

2. Formula
   Observation

   The board changes the heading to k-fold CROSS VALIDATION and lists the k-step procedure.

k-fold cross validation is the specific equal-partition version of cross validation described in the clip.

1. Audio
   Observation

   After describing the inherent trade-off, the speaker says this is where cross validation comes into the picture and then defines partitioning into k bins.

The trade-off in a single split motivates the need for the repeated holdout scheme used in k-fold cross validation.

1. Audio
   Observation

   The speaker says the key thing is to run this multiple times and then average the testing set performances.

2. Formula
   Observation

   The board lists the experiments first and then Average test results from those k experiments.

Averaging the fold-wise test results is part of the k-fold cross-validation procedure.

1. Formula
   Observation

   The speaker writes 'TRAIN/TEST' and '10-fold C.V.' side-by-side as column headers.

2. Audio
   Observation

   The speaker asks to choose between the 'static train test methodology' and '10-fold cross validation'.

The video contrasts the static train/test split methodology with k-fold cross validation as two different approaches to achieve evaluation goals.

1. Audio
   Observation

   The speaker introduces cross validation by name.

1. Cross validation

1. Audio
   Observation

   The speaker explains the dilemma of splitting data into training and testing sets.

1. Problem with a single train-test split
2. Training versus testing trade-off

1. Audio
   Observation

   The speaker describes partitioning into k bins and running k experiments.

1. k-fold cross validation
2. Averaging across folds

1. Audio
   Observation

   The speaker works through the 200-point, 10-bin example.

1. Equal-size bin example
2. Example: 200 points, 10 folds

1. Audio
   Observation

   The speaker says this takes more compute time because the experiment is run multiple times.

1. The sentence is incomplete at the clip boundary.

1. k-fold cross validation

1. Formula
   Observation

   The board lists the exact steps of the k-fold process.

1. Steps of k-fold cross validation

1. Formula
   Observation

   The board lists the three main goals for algorithm evaluation.

1. Goals in evaluating machine learning algorithms