Week 12: Calibration#

The first four words#

• predicted probability — the model’s stated long-run rate for an event among comparable cases
• binary event — an outcome with two classifications: success or failure
• bin — the cases assigned to one precommitted probability value or interval
• observed frequency — successes divided by cases in that bin

A probability is not a confidence adjective. It is a claim about repeated comparable cases.

Three more#

• reliability point — (predicted probability, observed frequency) for one bin
• reliability diagram — those points plotted against the identity diagonal, where prediction and observed frequency are equal
• domain of validity — the cases and mixture for which the evidence supports use, plus the cases where the probability should not be believed

The diagram needs its count table. A point alone hides how many cases produced it.

What the sources establish#

Guo et al. (2017) support the bin calculation, observed-frequency plot and identity-diagonal reference.

Hébert-Johnson et al. (2018) motivate checking calibration inside an identifiable subset rather than letting a coarse aggregate settle the claim.

They did not study this race fixture. Every record, probability, condition and result that follows is synthetic SlopU material.

Commit before outcomes#

Event: car 14 finishes P3 or better.

• P means classified finishing position; lower is better
• success: P1, P2 or P3
• failure: P4 or lower

Probability values: 0.20, 0.40, 0.60, 0.80

Named condition split: dry / wet

Freeze all three before revealing results. Moving a bin or choosing a condition after seeing outcomes lets the audit choose its own question.

No diagram before validation#

Reject the archive if it contains:

• a duplicate case ID
• a probability outside [0, 1] or outside the committed values
• a finish outside P1 and the archive’s lowest valid classified position
• a missing or unknown dry/wet label
• an empty aggregate or condition/bin cell

assert bins_after_unlock == bins_at_commit
if validation_errors: emit_no_diagram()

Invalid input does not become a plausible chart.

Aggregate archive#

Outcomes are revealed. Condition labels stay covered.

All 80 cases and every value in this table are synthetic.

Derive the 60% point#

The 0.60 bin contains 20 cases and 12 successes:

observed frequency = successes / cases

12 / 20 = 0.60

So its reliability point is (0.60, 0.60).

All four aggregate points:

(0.20, 0.20) · (0.40, 0.40) · (0.60, 0.60) · (0.80, 0.80)

Aggregate reliability#

Stop before the labels come off#

Every aggregate point sits on the diagonal. Each has 20 cases.

Write the strongest claim that plot supports.

Then write one claim it does not support.

The dry/wet split was committed before outcomes. Predict what must be true inside both conditions for the aggregate claim to survive unchanged.

Reveal the complete archive#

Counts are successes / cases.

Same fixed bins. Ten dry and ten wet cases in every bin. Every count is synthetic.

Dry condition#

Dry is above the stated probability at every value.

Wet condition#

Wet is below the stated probability at every value.

Stratified reliability#

Why the aggregate looked perfect#

At 0.60:

• dry: 8 / 10 = 0.80
• wet: 4 / 10 = 0.40
• aggregate: (8 + 4) / (10 + 10) = 12 / 20 = 0.60

The +0.20 dry error and −0.20 wet error cancel because every bin is exactly 50% dry and 50% wet.

Change that mixture and the aggregate frequency changes. The aggregate result belongs to the archive’s mixture, not to the model everywhere.

Draw the trust boundary#

The aggregate probability is supported only for the archive’s 50/50 dry–wet mixture. Do not use it as a condition-specific probability in dry-only or wet-only decisions until those strata are recalibrated and rechecked on new cases.

That is the domain of validity for this constructed archive.

Ten cases per condition and probability value expose the designed split. They do not remove sampling uncertainty. One realised race supports no calibration claim.

The report schema#

Record:

1. event definition and synthetic archive identifier
2. immutable probability bins and predeclared condition split
3. counts, successes and frequencies for every aggregate bin
4. aggregate diagram and archive-bounded finding
5. condition counts and stratified diagram
6. limitations, domain of validity and evidence needed for wider use

The plot is one part of the report. The trust claim is the result.

Two assertions that protect the audit#

assert event_after_unlock == event_at_commit
assert bins_after_unlock == bins_at_commit
assert condition_split_after_unlock == condition_split_at_commit

assert count == successes + failures
assert observed_frequency == successes / count
assert every_aggregate_and_condition_cell_is_nonempty

The first block protects the question. The second protects its answer.