Week 11: Defend the report
A number is not evidence. Declare the protocol behind a claim, run it twelve times, and find out what it will carry.
19 October 2026
The final report is not asked to find the true score. For the candidates on this bench there is no instrument-free answer to find: week 8 left two honest protocols ranking the same pair in opposite orders, and neither of them is the mistaken one. What the report is asked for is a claim a reader can check, which means a claim that carries the protocol that produced it.
That is a lower bar than a true number and a much higher one than a table.
Bring the evidence, including the gaps#
Bring the measurement log, a candidate reporting table, and your strongest reason for keeping FID. Separate what you measured from what you would still have to run. Week 9’s protocol belongs in that second list too.
Declare a claim#
The instrument below takes a claim and the evidence you are offering for it. Name which candidate you expect to score higher, name the instrument, name the sample size, and say whether the recordings have any temporal structure at all. Then it runs that protocol twelve times and shows you every run. The base seed is editable under Reproduce this experiment: rerunning the same settings and seed reproduces the same draws; choose a new seed for fresh recordings. Keep the seed fixed when comparing instruments on the same evidence.
The experiment summary starts with a conclusion and counts of the repeats that support the claim, contradict it, or give no ordering. The strip preserves each individual outcome. The comparison table shows all three instruments on the same recordings at the same N, so a reversal is visible beside the selected result. This is a calculated summary using fixed wording, not an AI-written report.
Try the claim you believe. Then try it through a different instrument without changing anything else.
What twelve repeats can and cannot settle#
Twelve repeats describe the stability observed in those draws. Even twelve agreeing verdicts do not prove zero sampling uncertainty or guarantee the next result. Report the observed counts, including mixed outcomes, rather than replacing them with a claim of certainty.
Set the frames to independent and run it again. Frame-to-frame scores often leave both candidates below the demonstration threshold. The panel then reports no ordering under that heuristic, not proof that the distributions are equal. The rule uses two baseline draws per repeat, just as week 8 does; its formula is typeset under Recorded protocol and numerical method and is not a calibrated test.
The Additional sample-size checks table adds one trial at each sample size on offer. Both its values and its heuristic verdicts can fluctuate. A trend across these individual trials is not an estimate of the trend in expectation; week 5’s distinction still applies. Retain the sampling conditions rather than interpreting a smaller number as automatically better evidence.
The conclusion and its conditions#
The experiment summary gives you evidence to interpret, not a finished student report. Its recorded protocol names the candidate, instrument, sample size, sequence length, temporal correlation, shift, base seed and repeat count. The method disclosure retains the heuristic; the comparison table retains all instruments’ counts, including reversals and mixed results. Use those conditions when writing your own conclusion and its limitations.
Notice what it never says: that the candidate is worse. Only that an instrument at a sample size, run a stated number of times, called it worse. The checker can be that careful because these candidates were built here and their differences are known. Your report has no such access, which is an argument for more care, not less.
Carry it into the defence#
Leave with an annotated reporting table and a named limitation rather than a claim that evaluation is solved. Prepare the case for comparability in the week 11 session, test your proposal against the final report requirements, and defend it at the week 12 presentations, where the bench gets scored one last time.
Claim → evidence → conclusion
What can this experiment support?
Choose which candidate you expect to score higher, then test that claim across twelve repeats. These are synthetic numerical sequences, not real images or videos.
- E / Level shift
- Every value is raised by 1. The sequence order is preserved.
- F / Reordered frames
- The values are preserved, but their order within each sequence is shuffled.
2. Experiment summary
A calculated summary using fixed wording, not an AI-written report.
Test a claim to see which conclusion the evidence supports.
which candidate each repeat scored higher
not run yet
| Instrument | E higher | F higher | No ordering |
|---|---|---|---|
| Run the experiment to compare instruments. | |||
These counts describe the observed repeats. They do not establish certainty or instrument-independent quality. “No ordering” is a heuristic outcome, not proof of equivalence.
Recorded protocol and numerical method
- Protocol
- Not run yet.
Demonstration threshold
b1 and b2 are two independent reference-against-reference scores for the selected instrument in that repeat. T is the demonstration threshold.
If both candidate scores are below T, the rule gives no ordering. If only one is below it, the other is ranked higher. Otherwise, the larger score is ranked higher; an exact tie is assigned to F by the current implementation.
This is a heuristic, not a calibrated statistical test. “No ordering” does not establish equal distributions.
Replaying the twelve repeats
sbase is the base seed; i runs from 0 to 11. ui is an intermediate value; si is the seed for that repeat. Modulo keeps it in the unsigned 32-bit range.
The numerical engine is bench.ts, using splitmix32 draws and sample covariance with denominator equal to the feature-row count minus one.
Seeds for the additional sample-size checks
Here N is 100, 400 or 800 sequences. XOR is bitwise exclusive OR; vN is the intermediate value and sN is that additional trial's seed. These checks are separate from the twelve repeats above.
Additional sample-size checks
One additional trial at each sample size, using the selected instrument. These individual trials do not establish a trend in expectation.
| N | Baseline 1 | Baseline 2 | Outcome |
|---|---|---|---|
| Not run yet. | |||
The protocol above could be run twelve times because every number in it was synthetic and every setting was declared. The activity below has neither advantage: one image comparison, on pixels you choose, measured once. What carries over is the order of work. Write the claim and the result that would contradict it before you measure, and keep every setting beside the number when you export it.
Supporting experiment / Workspace
An exported number still needs an argument.
01 Predict
Name one controlled pixel change and write what you expect it to do to the score. State what result would contradict your account.
02 Open the lab
Image lab / pixels to featuresRun an identity control and one altered candidate in Image lab, then export both measurement records.
03 Record
Keep the counts, encoder identity, preprocessing, candidate order and intervention with both records. Separate measured settings from assumptions.
04 Result
Draft two sentences: what the comparison showed, and whether it agreed with the prediction. Use the lecture’s reporting questions to defend those sentences.
05 Explain the limitation
An export is evidence, not a generated report or proof of human preference. These tiny image sets do not replace the final report’s shared-bench evaluation against FID and FID∞.
Keep this supporting experiment distinct from the task above. Workspace use and reporting guidance →
L11 / Optional self-check · Not assessed
Before you leave
No marks or penalties. Try it, retry it, or read the explanation. Nothing here is submitted.
All twelve repeats in the report panel support your declared ordering under the selected protocol. Which conclusion is defensible?
Answer and explanation
Correct: B. Report the observed count with the instrument, sampling settings and reproducibility information. Twelve agreeing observations describe those trials. They do not eliminate sampling uncertainty, validate the panel's heuristic as a significance test or settle what another instrument measures.
Feedback for A: A finite set of agreeing repeats is not a guarantee about the next draw.
Feedback for C: Week 8 showed why different feature maps can support different rankings on the same recordings.
Teaching takeaway: State the evidence and its conditions, not a stronger claim.
Collection code
SLOP-L11This code opens the matching comic. It is not a grade or proof of mastery. If browser storage is unavailable, copy the code and enter it on the collection page.