Week 12 Lab: Draw the trust boundary

Validate and audit a supplied probability archive, expose the condition split hidden by its aggregate diagram, and write the model's domain of validity

24 May 2027

Two hours, and you submit inside the session. This lab carries 2% of the course. You will audit one supplied model against its synthetic archive, draw its aggregate and condition-stratified reliability diagrams, and state where its probabilities should not be believed.

The event is car 14 finishes P3 or better. P means classified finishing position, with the lower number better. P1, P2 and P3 are successes; P4 or lower is failure.

The binary event, probability bins and condition split are frozen before the locked outcomes appear. You are not building or recalibrating a model. You are testing whether its stated probabilities occur at their stated rate.

Before the session#

Install the supplied codebase and run its tests. Read the case, event, bin and report schemas until you can name every required field. Classify the public finish examples as success or failure under the supplied event definition.

Write down the candidate probability bins and the one named condition split you will commit. A condition-redacted archive releases after validation; its condition labels and ground truth stay unavailable until the aggregate checkpoint. Arrive able to explain why moving a bin after seeing an outcome would make the audit answer a question chosen by its own results.

What is supplied#

Nothing in this panel is a download. The materials are not included in this prototype.

In the session#

1. Precommit and validate (25 minutes). Freeze the event, the probability bins and the named condition split. Reject duplicate case IDs, probabilities outside [0, 1], invalid classified finishes, missing or unknown conditions, predictions outside the plan and empty aggregate or condition/bin cells in the validation fixtures. Prove that any validation failure produces no diagram, then unlock the condition-redacted archive.
2. Build the aggregate check (25 minutes). For every fixed bin, calculate the case count, number of successes and observed success frequency. Plot predicted probability against observed frequency with the identity diagonal as the reference. Keep the count table beside the plot.
3. Interpret what it supports (15 minutes). Compare each aggregate point with the diagonal. Write one claim bounded to this archive and mixture. Do not turn agreement in an archive into a guarantee about a future case.
4. Reveal and stratify (30 minutes). Unlock the condition labels and ground truth. Repeat the same fixed-bin calculation inside both predeclared conditions. Explain how the conditional departures cancel in aggregate and identify the first condition-specific trust claim the evidence does not support.
5. State the domain and transfer it (25 minutes). Write the supplied model’s domain of validity: supported cases and mixture, unsupported uses, finite-sample limit, and the new evidence required to widen it. Emit the supplied-model report, then complete the blank plan for auditing your own instrument before Friday. Do not import that instrument here.

Checks your implementation must expose#

The precommitment test must fail if code changes a bin boundary or the named condition plan after unlock. The aggregation test must independently recompute each count, success total and frequency from validated records. A chart file is created only after every validation and non-empty-cell check passes.

The report records the event, archive identifier, immutable bin plan, counts, both diagrams, condition split, limitations, domain statement and evidence needed for wider use. Every record and result in it is labelled synthetic.

What you hand over#

• validators and tests for the frozen audit plan;
• aggregate and condition-stratified count tables, with both reliability diagrams;
• an archive-bounded finding and the supplied model’s domain of validity;
• a separate blank-plan response for auditing your own instrument before Friday.

Only the supplied-model audit is marked in this lab. The final instrument later applies the operation to your own probabilities and asks for the domain of validity you are willing to declare.