Week 8 Session: Ask the Room

Two rounds of estimates from the same people under two different protocols, and a national polling failure that has the same shape as the second round

26 April 2027

Today the room is the dataset. Two rounds, written not spoken, and no phones until the reveal.

Protocol#

Both quantities are published monthly by the Bureau of Infrastructure and Transport Research Economics. Both are settled facts sitting in a spreadsheet; neither is a prediction, and nothing today asks anyone to guess at the future. The only thing that varies between the rounds is what you are given to work from.

Round A. Estimate the total number of passengers through Hobart Airport in August 2025. You get the question and nothing else. Write your figure on a slip, fold it, hand it in. No discussion, no comparing, no adjusting after you have written.

Round B. Estimate the total number of passengers through Gold Coast Airport in August 2025. Everyone is given the same starting point, and it is a real published figure from the same BITRE series:

Gold Coast Airport, August 2021: 12,485 passengers.

Reason from it however you like. Same rules: written, folded, handed in, no conferring.

Collect both rounds before anybody says anything.

Diary note. Before you fold and hand in each slip, copy that estimate into your Calibration Diary as a full entry: a low bound and a high bound around your figure, your confidence in plain words, a miss-line, and the BITRE airport traffic series named as your resolution source. This is the Diary’s instruction, not a change to Round A or Round B themselves.

The reveal#

Truths, from the BITRE workbook:

• Hobart, August 2025 — 207,990 passengers.
• Gold Coast, August 2025 — 505,540 passengers.

Put every Round A slip on the board and find the middle. Do the same for Round B. Two aggregates, two spreads, computed in front of everyone.

Then look at the shape of each round, not just its accuracy:

• Where did each aggregate land relative to the truth?
• How wide was each round’s spread? Which round was tighter? Whatever the answer, do not draw comfort from a tight spread until you know why it is tight.
• Which direction were Round B’s misses? Scattered on both sides of the truth, or bunched to one side? Count them before anyone offers an explanation.

What went wrong in Round B, and what did not#

Nobody was lied to. 12,485 is exactly what BITRE published for Gold Coast in August 2021. It is a real number from a reliable agency about the correct airport in the correct month of the year.

It is also the wrong number to reason from, for a reason that is obvious once somebody says it and easy to miss when you are working: August 2021 fell in a period of domestic border closures. The Gold Coast handled around twelve thousand passengers that month and around half a million four years later — a factor of roughly forty. Anybody who took the reference seriously and scaled it by a plausible-sounding recovery multiple landed far short.

Ask, by show of hands, how many spotted the problem and adjusted — and then look at where those estimates landed against the ones that did not. Adjusting from a low base can still leave you low, because the base sets the scale you are adjusting from; whether it did so here is something your own slips can answer.

That is the mechanism to test against what you just measured. Nothing about Round B asked for less care than Round A. If its estimates came in bunched and low anyway, the explanation is not effort: everyone reasoned separately from one shared input, and an error in a shared input passes straight through every chain into every answer. Averaging removes the parts of an error that differ between people. It cannot touch the part they all hold in common, and adding more people to the room does not help.

So check the two readings against each other. If the tighter spread was also the one further from the truth, then agreement — which reads as confidence — was at its highest exactly where the room was most wrong.

The same shape, at national scale#

In May 2019 every published poll had Labor winning the Australian federal election. The Coalition won. An inquiry was commissioned by AMSRO and the Statistical Society of Australia; its report came out in October 2020. Its finding:

“The performance of the national polls in 2019 met the independent criteria of a ‘polling failure’ not just a ‘polling miss’. The polls: (1) significantly—statistically—erred in their estimate of the vote; (2) erred in the same direction and at a similar level; and (3) the source of error was in the polls themselves rather than a result of a last-minute shift among voters.”

This was not a run of bad luck by an unreliable industry. The same report:

“In the four elections preceding the 2019 election – 2007, 2010, 2013, and 2016 – there was a 96% success rate (25 correct calls out of 26 polls).”

The obvious suspicion is that the pollsters copied each other. The panel looked and could not establish it:

“While the Panel found no direct evidence of herding, this cannot be ruled out”

“In the absence of any direct evidence of herding, the most likely explanation is a systematic skew in the polls coupled with a confirmation bias.”

The skew:

“even though the methods used by the pollsters differ they share a common difficulty in struggling to establish contact with and gain the cooperation of a representative sample of voters.”

“the kinds of people who are willing to participate in polls skew towards the more highly educated/more engaged in politics”

Separate companies. Separate samples. Separate methods. One difficulty they all had, which pushed all of them the same way — and, because the results agreed, looked from outside like several independent confirmations of the same finding.

Read that against your Round B slips. The mechanisms are not the same — you were all handed one number, while the pollsters had separate samples and a shared difficulty in reaching them — but the failure mode is: error common to every estimate survives aggregation, so agreement between them is not independent confirmation.

Twenty minutes on your own work#

In small groups, go back through the estimates you have made this semester: the week 2 chain, the week 3 ledger, the week 4 ranges, the bridge from week 5.

Find one where your inputs were not yours alone. The same lecture, the same dataset, the same way of breaking a problem into parts that week 2 taught all of you, the same starting figure everybody in the room happened to know. Name it.

Then say what you would have to do to get a second estimate that did not share it. This is usually harder than it sounds, and noticing that it is hard is the point.

Leaving#

Nothing is due. Keep the week 4 sheet going.

One thing to carry into the last stage: for the rest of the semester, when a set of estimates agrees, the first question is no longer whether they are right. It is what they have in common.