Week 12: One miss tells you nothing. Fifteen might.

Why no single estimate can reveal whether you are a good estimator, and what an accumulated, precommitted record can show instead

24 May 2027

Open the slides

The object being studied changes today. Until now, each week has asked whether one particular number was any good — not the Melbourne count, the airport temperature, or any other quantity out there in the world. Today it is you, across everything you have written down since week 4.

What a single result cannot carry

A forecaster who says 70% and turns out right has told you almost nothing about their judgment. A forecaster who says 70% and turns out wrong has told you almost as little. One result is compatible with excellent judgment, terrible judgment, and pure chance all at once, because a single outcome cannot distinguish between “this was well-calibrated and simply landed in its losing 30%” and “this confidence was invented.” Whether a single range held is barely evidence about the person who wrote it.

What an accumulated record can carry

Fifteen results are a different kind of object. A record that says 75% confidence again and again, and holds the true value close to three times in four, is showing you something a single entry structurally cannot: a consistency between stated confidence and what actually happened, repeated enough times that chance stops being a plausible explanation on its own. This is the entire logic behind the Bureau of Meteorology checking its own forecast accuracy against outcomes at a national scale rather than trusting any single day’s forecast to prove anything — the pattern lives in the accumulation, and nowhere else.

Notice what this does and does not require. It does not require a formula. Whether an assumption survives being said aloud in week 11 or a range holds in week 10, the check is the same plain act: look at what happened, compare it with what was claimed, and count. Pattern-finding is counting done across many entries instead of one; nothing about the arithmetic changes.

The line from week 4 was pointing here

Week 4 closed by telling you to keep the sheet, because the line naming what would make your range miss “turns out to be worth something in week 12.” This is that week. A single miss-line, written before you looked, is a private prediction about your own judgment. Fifteen of them, read together, are the only way to find out whether that prediction was accurate about you — whether you tend to be wrong for the reasons you named in advance, or for some other reason you never once wrote down.

Turning the method on its author

Everything this course has taught about a number applies, unchanged, to a person’s set of numbers: name the pattern, then actively look for the entry that breaks it, the same discipline week 3 asked of a single assumption and week 9 asked of a single scheme’s ledger. What is different is only the target. For eleven weeks the target was always something out there — a bus stop, a dwelling count, a scheme, a cash rate. Today it is the habits of the one person who has been doing the estimating the entire time, and the record that makes that visible had to be built weeks before anyone needed it.

Outline