Week 8: Change the instrument

Two honest numbers from the same recordings can disagree about which candidate is worse. Put three instruments on one bench and find out why.

28 September 2026

Try it in the workspace →

Optional self-check ↓

A score is never a measurement of the thing. It is a measurement of the thing through an instrument, and the instrument decides what counts as a difference before any arithmetic happens. Week 3 made that argument about Inception. This week it stops being an argument, because two instruments on this page disagree about which of two candidates is worse, at the same sample size, from the same recordings, with the same formula applied to both.

Nothing has gone wrong when that happens. That is the difficulty.

Bring a testable failure

Bring one temporal failure you expect a video metric to miss, and the controlled comparison that would prove you wrong. A convincing pair of clips is a hypothesis about an instrument, not yet a measurement of one.

Three instruments, one bench

The bench below is not video. It is the one property of video this lecture needs: sequences whose neighbouring frames are related, so that the order of the frames carries information. Set how many frames each sequence has, how many sequences are drawn, and how strongly neighbouring frames are correlated. Each candidate starts from an independent draw of R’s process. E raises every frame by the same amount and leaves the order alone. F permutes the frames within each sequence. It preserves the values in its own draw, not those in the separate reference set. Each scored set is reused across the three instruments, and the plot shows its first five sequences.

Then three instruments score them. Frame marginals pools every frame of every sequence and fits one distribution to the values, which is what a frame-wise measure sees. Frame to frame scores the differences between consecutive frames. Whole sequence takes all the frames jointly, which is what the bench has done since week 1.

Read the first row first

The first row scores R against a second independent draw of R. Although the population distributions match, sample estimates generally do not report zero. Read this baseline before comparing the candidate scores; it is a random baseline, not a lower bound on every possible score.

The labels use two independent baseline draws and an explicitly stated demonstration threshold. A cell below that threshold is not flagged by the heuristic. That is not a significance test or proof of blindness.

There are also exact properties we can establish from the construction: pooling frame values discards their order, and taking consecutive differences removes a constant shift. These operations lose information regardless of where a particular finite-sample score lands.

The disagreement

Before you run it, write down which of E and F you expect each instrument to call further from R. Then press Measure all three instruments at the default settings and record, for each instrument, the two scores and which candidate it ranks higher. Record the actual scores, not only the order; another draw will not repeat them exactly. Compare them with your prediction, then open the fold.

Expected results

Frame to frame should rank F higher, and whole sequence should rank E higher. Expect that direction, not a fixed ratio or the same verdict on every draw. The same recordings can support different rankings because the feature maps preserve different properties before the Gaussian formula is applied.

Set the correlation to zero and run it again. Permuting independent, identically distributed frames preserves their population distribution. Finite samples can still produce nonzero scores and occasional flags; the population invariance is not a promise about every displayed verdict.

Carry it forward

Record which instrument produced each number in your measurement log, beside the sampling protocol week 5 asked for. Work the reading and the exercise in the week 8 session. The week 11 lecture asks whether your final report claims more than a named instrument at a named sample size can support.

Record → transform → compare

Does the ranking survive a different instrument?

Score the same sets of synthetic numerical sequences three ways. These sequences model temporal structure; they are not real images or videos.

E / Level shift
Add 1 to every frame value, preserving sequence order.
F / Reordered frames
Shuffle the values within each sequence, preserving the values themselves.

1. Set the recordings

One set of recordings, three instruments, nine numbers.

2. Inspect what changed

showing
Press measure to draw the first five recordings from the scored sets.

3. Compare the measurements

Press measure to compare the three rankings.

Each candidate's recordings are reused across all three columns. The first row compares independent draws from R. Labels use a classroom heuristic, not a significance test or proof that an instrument is blind.

Nine scores, one set of recordings.
what changedframe marginalsframe to framewhole sequence
nothing, R drawn again   
E, a level shift   
F, frames reordered   
What each instrument measures
Frame marginals
Pools individual values, discarding their order.
Frame to frame
Uses consecutive differences. A constant shift cancels.
Whole sequence
Uses all coordinates jointly, retaining their relationships.
Δxt=xt+1−xt

Here xt is a frame value, t its position and Δxt the difference to the next frame. Each instrument fits a Gaussian to its resulting features.

How the demonstration threshold works

Demonstration threshold

T=1.6⋅max(b1,b2) +4|b1−b2|+0.001

b1 and b2 are two independent reference-against-reference scores for the selected instrument in that repeat. T is the demonstration threshold.

If both candidate scores are below T, the rule gives no ordering. If only one is below it, the other is ranked higher. Otherwise, the larger score is ranked higher; an exact tie is assigned to F by the current implementation.

This is a heuristic, not a calibrated statistical test. “No ordering” does not establish equal distributions.

The bench above shuffled frames inside each sequence, and only an instrument that reads order could tell. The activity below shuffles something else: the order of whole images in a collection, in the image lab, with real Inception features. Predict what that shuffle can change before you press it, and keep the two kinds of order apart when you bring the result back to the pooled frames and the whole sequence.

Supporting experiment / Workspace

Collection order and temporal order are different questions.

  1. 01 Predict

    Will reordering candidate images change collection FID, the paired feature distance, or both? Which prediction would require a temporal feature extractor instead?

  2. 02 Open the lab

    Image lab / identity and shuffle

    Measure the supplied JPEG comparison, then shuffle and restore candidate order without changing any pixels.

  3. 03 Record

    Export the original and shuffled ordering with the same feature vectors, counts and preprocessing.

  4. 04 Result

    Compare both displayed quantities before and after shuffling. Bring that distinction back to the lecture’s pooled-frame and sequence instruments.

  5. 05 Explain the limitation

    This lab compares unordered image collections, not videos. Image-order invariance does not establish that FVD ignores frame order; temporal sensitivity must be tested with the actual video features.

Keep this supporting experiment distinct from the task above. Workspace use and reporting guidance →

L08 / Optional self-check · Not assessed

Before you leave

No marks or penalties. Try it, retry it, or read the explanation. Nothing here is submitted.

Take one recorded sequence x and construct y by adding the same constant c to every frame: yₜ = xₜ + c. An instrument keeps only consecutive differences, Δxₜ = xₜ₊₁ − xₜ. What happens to its features?

Answer and explanation

Correct: C. Δyₜ = (xₜ₊₁ + c) − (xₜ + c) = Δxₜ. This exact construction shows information discarded by this particular feature map. More samples cannot recover the constant from features that removed it.

Feedback for A: Both adjacent frames receive c, so subtracting them cancels it.

Feedback for B: Sample count affects estimation, not this algebraic invariance.

The lecture panel uses independent reference and candidate draws. Its finite-sample scores need not be zero even when the feature distributions match. This question compares a sequence with its own shifted copy, not two independent recordings, and makes no claim about I3D or actual FVD.

Teaching takeaway: An exact calculation can still operate on incomplete information.