Week 8: What your estimates have in common

The single mechanism that makes combining estimates work, what happens when it is absent, and why agreement is not evidence

26 April 2027

Open the slides

Combining estimates is the most reliable trick in this course, and it works for exactly one reason. Understand the reason and the failure mode follows from it without needing to be memorised separately.

One mechanism

Ask ten people and the errors point in different directions. Some read the situation too high, some too low, for reasons particular to each of them — what they happened to have seen, which comparison came to mind, which part of the problem they thought mattered. Put the estimates together and those differences work against each other. What survives is closer to the truth than most of the individuals were.

That is the whole of it — errors that differ partly cancel. Whatever else gets said about the wisdom of crowds, that arithmetic is doing all the work.

So the question to ask of any set of estimates is not how many there are. It is whether their errors differ. Ten estimates that all lean the same way are worth about as much as one, and cost ten times as much to collect.

What survives averaging

Split the error in any one person’s estimate into two parts: the part specific to them, and the part they share with everyone else in the room.

Combining works on the first part and does nothing whatever to the second. If everyone started from the same reference figure, everyone read the same briefing note, everyone learned the technique from the same person, or everyone is reachable by the same recruitment method, then whatever is wrong with that common input is present in every estimate at full strength and comes through the average untouched.

No amount of additional estimates helps, because each new one arrives carrying the same error. This is the counter-intuitive part and the part worth holding onto: more data does not dilute a shared mistake. It reproduces it.

What tight agreement tells you

There is one further property, and it is the one that does the damage.

Independent estimates spread out. Estimates sharing an input cluster. So the signal you are most likely to read as reassuring — everyone came out about the same — is exactly the signal that correlation produces.

Tight agreement means one of two things: the room converged on the truth, or the room shared something. From inside, the two look identical. You cannot tell which by staring harder at the estimates, because the evidence that would distinguish them is not in the numbers at all; it is in how they were produced.

The 2019 Australian polls are the cleanest available demonstration. Five final polls, unusually close to one another, all wrong in the same direction. The inquiry that followed could find no evidence that anybody had copied anybody. Its explanation was that the pollsters shared a difficulty — the people who agree to answer polls are not a cross-section of voters — and that this difficulty pushed every one of them the same way. The convergence that would ordinarily have signalled a well-measured result was produced by the thing that made the result wrong.

Auditing independence

Since agreement cannot tell you, the check has to be procedural. Before treating a set of estimates as several pieces of evidence rather than one, ask what they have in common:

Did they start from the same figure? Use the same dataset, or two datasets derived from a common source? Come from people trained the same way, or by each other? Get produced after the group had already discussed the problem? Reach their subjects through the same channel?

A yes to any of these does not invalidate anything. It tells you how much independent evidence you actually have, which is usually less than the count of estimates suggests, and occasionally a great deal less.

There is no formula here and this course will not give you one. The question is qualitative, it is answerable in a couple of minutes, and asking it at all puts you ahead of most rooms.

The uncomfortable version

Everything above applies to groups you are not in. It also applies to the one you are.

Your own estimates this semester were made after the same lectures, using the same worked examples, with the same handful of reference points in mind. Where they agree with each other, some of that agreement is real and some of it is the course. Distinguishing the two is not something anybody can do for you, and it is a reasonable thing to be uncertain about.

Outline