Week 9: Accurate on average is not the same question

Why expected accuracy stops being the point once an estimate gets acted on, and the two questions the rest of the course has kept merged

3 May 2027

Open the slides

Up to now, every improvement this course has offered has been an improvement to the number itself: a wider range, a named assumption, a disclosed shared input. All of that assumes the only thing worth asking about an estimate is how good it is. Today argues that assumption breaks the moment the estimate is used to act on somebody.

Two questions, one habit of merging them

The first question is the one the whole course has trained: how uncertain is this number, and have you said so honestly? Everything from week 4’s ranges to week 8’s independence check answers a version of it.

The second question is different in kind: once this number is acted on, who absorbs the gap if it is wrong? That question has nothing to do with how the estimate was built. It is about what happens downstream of using it, and it has an answer regardless of whether the estimate was careful or careless, well-calibrated or not.

The habit worth breaking today is treating the second question as if answering the first one had already covered it. A perfectly calibrated estimate can still be acted on in a way that puts its entire error onto the person least able to contest it. Calibration and consequence are orthogonal, and a course that only taught the first would have taught half the subject.

Why expected accuracy is not enough

An estimate that is right on average, across every case it is applied to, can still be a bad basis for individual action if the two directions of its error are not treated the same way afterward.

Picture two mirror-image errors of identical size and identical probability. If both are corrected by the same process, with the same ease, borne by the same party, then the average is the whole story and nothing further needs saying. If one direction triggers an automatic, enforced consequence and the other triggers nothing at all, the average has stopped describing anything that matters. The estimate can be exactly as good in expectation and the situation can be entirely different, because what changed was never the estimate — it was who has to do something when it misses, and in which direction.

This is why “how wrong can you be, and still be useful” cannot be answered by the estimate’s error properties alone. Useful to whom, and at whose expense when it is wrong, are part of the same question, not an afterword to it.

Institutional power changes who has to argue

There is a structural reason this asymmetry recurs rather than being a one-off accident. Whoever builds the estimate that gets acted on usually also controls the process for contesting it — the record of what was assumed, the channel for raising an objection, the resources to defend a challenge. The party the estimate is used against typically has none of that by default. So even before anything goes wrong, the two sides do not start from the same position: one side’s assumptions are backed by an institution, the other side’s rebuttal has to be built from nothing, after the fact, under a deadline it did not set.

That imbalance does not require bad faith on anyone’s part. It is present the moment an estimate produced by one party is used to make a claim against another, and it decides who bears the burden of proof long before either direction of error has occurred.

What this changes about how you write a number

Nothing about how you build a range changes. What changes is a question you now have to ask before you publish it: if this number is used and it is wrong, in each direction, who has to act — and can they?

An estimator who cannot answer that has not finished the job description this course has been building toward, however tight the range or however honest the confidence.

Outline