Week 1: The bench, live

The week 1 exercise as an instrument you can run in this page. Draw your own samples, score them, and watch the estimate disagree with the truth.

3 August 2026

Optional self-check ↓

The session tells you to score A and B against R and record what you get. This page is the same exercise with the waiting removed, on a bench small enough that your browser is the laptop.

Two controls set the candidate. The mean shift moves every coordinate of its mean, and the covariance scale v widens every coordinate of its variance. The other two set the measurement: d is how many dimensions the bench runs in, and N is how many samples each press draws from R and from the candidate.

Press score and read two numbers. The true FID comes from the closed form, straight from the parameters, with no sampling anywhere in it. The estimated FID is computed from the N samples you just drew, the way every published score is computed. Press draw again and the estimate moves while the truth does not. Raise d toward 64 and the two stop being on speaking terms.

Everything here runs in your browser, through an implementation pinned against the course’s reference implementation, and nothing you press leaves the page. Why the gap behaves the way it does is week 5’s whole subject; this page only lets you meet it early.

R
N(0, I) in d dimensions. The reference every candidate is scored against.
A
N(0.05·1, I). Every coordinate’s mean shifted by 0.05, variance untouched. True FID 0.160 at d = 64, and 5.12 at the bench’s 2048.
B
N(0, 1.1·I). Mean exact, every coordinate’s variance 1.1. True FID 0.152 at d = 64, and 4.88 at 2048.
C
Week 6’s candidate. R’s mean and covariance, but the first coordinate splits into two lumps at −0.9 and +0.9. True FID 0.
2 of 64 coordinates. The difference you cannot see is the whole of the number.
true FID, closed form
 
estimated FID, this draw
 

Every press adds a point. This is week 5’s figure, built by your own clicks. Point at a dot to read its estimate. Changing any setting clears it, because points are only comparable when every press measured the same thing.

L01 / Optional self-check · Not assessed

Before you leave

No marks or penalties. Try it, retry it, or read the explanation. Nothing here is submitted.

Keep the candidate's mean shift, covariance scale, dimension and sample size fixed. You press Draw again and the estimated score changes. Which statement is justified?

Answer and explanation

Correct: B. The population score comes from the fixed distribution parameters. The estimate uses the moments of the samples just drawn, which can differ between draws. A changing estimate alone does not demonstrate a change in the candidate or in the implementation.

Feedback for A: The candidate parameters were held fixed. Distinguish a new measurement from a new generator.

Feedback for C: Sampling variation can produce this observation without any implementation change. It is not by itself evidence of a software defect.

Teaching takeaway: The number can move when the thing being measured does not.