Week 12: Calibration

Backtest a supplied model's P3-or-better probabilities in aggregate and by condition, then state exactly where they should not be believed

24 May 2027

Open the slides

Week 11 put one probability in the largest type on the wall. This week we make a probability like that earn its space, using a separate supplied archive and event. A predicted probability is a claim about repeated comparable cases, not a confidence adjective and not a promise about one race.

We define the binary event before looking at results: car 14 finishes P3 or better. P means classified finishing position, with the lower number better. P1, P2 and P3 are successes; P4 or lower is failure. We then group archived cases by the probability the supplied model stated. A reliability diagram plots predicted probability on the horizontal axis and observed success frequency on the vertical axis. Agreement sits on the identity diagonal, where the two values are equal. Counts stay beside the plot because a point based on a small archive does not carry its sample size on its face.

The first view is reassuring. In SlopU’s constructed synthetic archive, every aggregate point lands on that diagonal. That supports one claim: the stated probabilities agree with the observed frequencies for this archive and its mixture of cases. It does not prove agreement in future races.

Then we apply the dry/wet split we named before seeing outcomes. The same fixed bins miss in opposite directions in both conditions. Dry departures and wet departures cancel in the aggregate because the archive contains the same number of each in every bin. The aggregate plot was correct and the broad trust claim was wrong.

We finish with a domain of validity: a precise statement of the cases and mixture for which the evidence supports using a probability, and the cases in which it should not be believed. The lab builds that statement for a separate supplied model from a locked synthetic archive. It validates the archive before drawing, reports counts with both aggregate and condition-stratified diagrams, and names what new evidence would justify wider use.

The final instrument asks you to run the same method on your own stated probabilities before Friday and submit your own declared limits. The lab’s assessed output is the supplied-model audit and its domain statement. You also leave with a blank audit plan for the transfer. Your instrument is not imported or marked in this lab, and the supplied-model domain is not submitted as the instrument’s domain. Week 11’s wall configuration is different: that artefact transfers literally and runs unchanged against the instrument’s shared output schema.