Why is pure random sampling inefficient in high dimensions, and how does MCMC solve this?

Conditions

• The target distribution is defined over a high-dimensional space.
• As dimensionality increases, random sampling becomes inefficient.
• MCMC uses a Markov chain designed so that its stationary distribution is the target distribution.

Reasoning, step by step

1. Identify the problem: in high dimensions, pure random sampling becomes nearly impossible to get good samples.
2. Recognize the cause: the volume of the space grows exponentially.
3. Introduce the solution: marry Monte Carlo with Markov chains to create MCMC.
4. Explain the mechanism: MCMC replaces blind randomness with guided exploration via dependent sampling.
5. Conclude the benefit: samples naturally concentrate in regions of high probability.

Example

The video states 'But in high dimensions... Random sampling becomes inefficient!' and then shows an animation of a red dot proposing moves on a bimodal distribution, illustrating how MCMC explores high probability regions.

Common misconceptions

• Assuming that increasing the number of random samples will eventually solve the high-dimensional sampling problem efficiently.
• Believing that MCMC samples are independent like ordinary Monte Carlo draws.

Watch the explanation

Explore next

• What is the key insight of dependent sampling in MCMC?
• How does the Monte Carlo method estimate the expected value of a distribution?
• What is a Markov chain and what is its memoryless property?