Week 5: Measured well, about something else

Why most evidence is borrowed, the three ways a proxy fails, and why a proxy relationship can decay without anybody noticing

22 March 2027

Open the slides

You will spend your career using numbers that were collected by someone else, for a purpose that was not yours, using a definition you did not choose. This is the normal condition of evidence rather than a regrettable exception, and a practitioner who refuses to work with it will answer almost nothing.

The bridge nobody writes down

When a dataset measures the thing you care about, there is no gap to think about. That case is rare enough to be worth noticing when it happens.

Ordinarily what you have measures something adjacent, and between the two runs a short chain of claims: this counts that, that tracks the other, the difference between them is small enough for the purpose. Each link is a statement about the world, and each could be false.

The trouble is that these claims are made silently. Nobody writes “I am assuming grid supply approximates total consumption” — they simply plot the series and talk about electricity. The bridge is crossed without being inspected, and often without being seen.

Three failures worth telling apart

They call for different responses, which is why lumping them together as “the data was bad” wastes information.

The bridge was never sound. The proxy and the target were only ever related by hope. Nothing about better measurement repairs this; you need a different source, and the sooner you conclude that the better.

The bridge was sound and the world moved. This is the interesting one. A relationship that genuinely held in 1995 can fail by 2025 without a single methodological change, because the term in the middle — the thing that made the two quantities close — grew. The data is as good as it ever was. The inference built on it is not.

The bridge holds, but the gap is too wide for this decision. Sometimes you know the proxy runs low by roughly a fifth and the decision turns on a difference of two per cent. The honest move is to say what the proxy supports and what it does not, rather than to pretend the gap away or to abandon a usable source.

Decay is quiet

The second failure deserves more attention than it gets, because of how it presents.

A series that has stopped meaning what it used to mean carries no signal that this has happened. It arrives on the same schedule, from the same agency, formatted the same way, with the same name at the top of the column. Its authority is unchanged. Nothing in the file records that the sentence you used to be able to say about it is no longer true.

The only defence is to have written the bridge down while it still held, with the load-bearing claim marked, so that there is something to re-read. An assumption you never articulated cannot be reviewed, and you will not spontaneously remember to question a step you never knew you took.

This is also the reason today’s session ends by asking for a condition rather than a caveat. “Treat with caution” is not reviewable. “This breaks once more than about a fifth of supply is behind the meter, and I would see it first in the summer minimum” is a thing you can go and check next year.

Where the good sources help

Publishers are often more honest about this than the people quoting them. AEMO maintains a definition sheet whose purpose is to stop you reading its numbers as something they are not, and it says outright that its measure of demand falls as rooftop generation rises. That sentence is doing real work, and almost nobody who cites the series has read it.

Read the definitions. They are short, they are boring, and they contain the information you need to know whether the number in front of you can answer your question. It is the cheapest form of diligence available and the most commonly skipped.

Outline