SLOP8412 / Optional collection / 17 teaching comics
Small comics.
Same hard questions.
A correct calculation can still support the wrong conclusion. GPT and Claude meet seventeen ways that happens, one short conversation at a time.
Use a code from a lecture or session self-check. Codes already unlocked in this browser appear automatically. This is ungraded, with no account or submission.
Reset all comic unlocks in this browser?
This also resets the matching quiz unlock flags and ends Spoilers. Your theme, workspace and lab settings are untouched. You can unlock comics again with the same codes.
Lectures0 / 5 collected
L01 / Week 1A new draw. Not a new model.Locked
Find this card's code in the optional self-check for The bench, live. You can solve the question or review its explanation.
Read transcript
GPT makes tea while Claude draws fresh samples at the laptop. The estimated score changes although the model stays fixed. GPT drinks her tea with both hands as they identify what actually changed.
- GPT: I’ll make tea. Keep the model fixed.
- Claude: New samples. Same model.
- Claude: The estimated score changed!
- GPT: Did my generator improve while I made tea?
- Claude: The samples changed. The population didn’t.
- GPT: So the tea gets no credit.
Teaching labels: Candidate parameters: unchanged · New samples → new empirical moments · Population score: unchanged
Keep this: The number can move when the thing being measured does not.
The model parameters stay fixed while new samples change empirical moments. The illustration is not a measured experiment.
L02 / Week 2The missing termLocked
Find this card's code in the optional self-check for The Fréchet distance. You can solve the question or review its explanation.
Read transcript
Claude checks her handwritten calculation beside the laptop. GPT, holding coffee, points out the omitted spread term. Claude scratches her head and adds one to the four, giving five.
- Claude: The means are two apart. Squared distance: four. Done?
- GPT: The standard deviations are one apart too. That contributes another one.
- Claude: Five. My calculation left the spread outside.
Teaching labels: Mean term: (0 − 2)² = 4 · SD term: (1 − 2)² = 1 · D = 4 + 1 = 5
Keep this: A location mismatch and a spread mismatch contribute separately.
This is the stated 1D Gaussian population example. The second term uses standard deviations, and D is the squared distance.
L05 / Week 5Precisely over thereLocked
Find this card's code in the optional self-check for Question the estimate. You can solve the question or review its explanation.
Read transcript
GPT celebrates a tight cluster around seven. Claude presses Enter to show the population value of five. GPT recognises the bias; they share cookies. The displayed values and plots illustrate the example, not measured results.
- GPT: More independent repeats! Our average is settling beautifully around seven.
- Claude: Beautifully. The population value is five.
- GPT: A very tidy cluster in the wrong place.
Teaching labels: Fixed N · estimator expectation = 7 · Population value = 5 · bias = 2 · Less variation ≠ less bias
Keep this: Precision and accuracy answer different questions.
At the same N, averaging independent estimates reduces the variance of the average, not its bias. Individual averages need not improve monotonically.
L08 / Week 8The vanishing shiftLocked
Find this card's code in the optional self-check for Change the instrument. You can solve the question or review its explanation.
Read transcript
Claude adds ten to each frame. GPT writes the consecutive-difference equation to show the two tens cancelling. Surprised, Claude holds up a board saying the difference features are identical.
- Claude: I added ten to every frame. Surely the instrument will notice.
- GPT: This one keeps only consecutive differences. Your two tens cancel.
- Claude: Ten arrived twice and left no forwarding address.
Teaching labels: yₜ = xₜ + 10 · (xₜ₊₁ + 10) − (xₜ + 10) = xₜ₊₁ − xₜ · Identical difference features
Keep this: An exact calculation can still operate on incomplete information.
An exact property of this toy feature map, not a claim about actual FVD. Independent reference and candidate draws can still give different sample estimates.
L11 / Week 11Twelve is a countLocked
Find this card's code in the optional self-check for Defend the report. You can solve the question or review its explanation.
Read transcript
A puzzled GPT asks whether twelve agreeing repeats justify certainty. Claude corrects the claim and fetches tea. GPT has a seatbelt-not-crown realisation; Claude passes her a cup and they drink together.
- GPT: Twelve out of twelve repeats agreed. May I write “certain”?
- Claude: Write “twelve out of twelve”, with the protocol. The thirteenth has not happened.
- GPT: The conclusion gets a seatbelt, not a crown.
Teaching labels: Observed support: 12/12 · Instrument · N · seed · repetitions · Evidence with conditions
Keep this: State the evidence and its conditions, not a stronger claim.
The panel’s heuristic is not a calibrated significance test. Agreement in these repeats does not guarantee future agreement or agreement between instruments.
Sessions0 / 12 collected
S01 / Week 1The bold numberLocked
Find this card's code in the optional self-check for The number. You can solve the question or review its explanation.
Read transcript
Claude proudly shows a table with 4.8 in bold above 5.2. GPT asks about the protocol. Claude shakes her head, admitting the comparison is not ready. GPT nods with a lightbulb above her head. The scores are illustrative.
- Claude: This row has the lower FID. I have already made it bold.
- GPT: Same reference set, N, features, preprocessing and implementation?
- Claude: The bold font is ready. The comparison is not.
- Claude: You should check conditions before celebrating.
- Closing board: Comparable conditions before bold conclusions.
Teaching labels: 4.8 versus 5.2 · Protocol: not yet checked · Check conditions before celebrating
Keep this: Ask what produced the number before reading the ranking.
The scores are illustrative. Compatible conditions do not remove estimator bias or uncertainty, or establish overall model superiority.
S02 / Week 2Perfectly ordinary, separatelyLocked
Find this card's code in the optional self-check for The Fréchet distance. You can solve the question or review its explanation.
Read transcript
GPT compares matching marginal curves. Claude takes the laptop and shows joint contours: a circle for the reference and a tilted ellipse for the correlated candidate. GPT types her corrected conclusion while Claude watches with tea.
- GPT: Both coordinate curves match. The distributions must match too.
- Claude: Look at them together. One pair of coordinates is correlated.
- GPT: I interviewed each coordinate separately and missed the relationship.
Teaching labels: Each marginal: Normal(0, 1) · R: I · G: [[1, 0.8], [0.8, 1]] · Zero marginal distances · positive joint distance
Keep this: Looking at coordinates separately can hide a relationship.
These are zero-mean 2D Gaussian population distributions. The joint covariance term retains information that separate coordinate curves omit.
S03 / Week 3More digits, same blind spotLocked
Find this card's code in the optional self-check for The instrument. You can solve the question or review its explanation.
![More digits, same blind spot. Claude inspects changed images whose displayed feature vectors both equal [1, 2]. GPT removes her reading glasses to explain the missing information. Claude shuffles the images, then hums after recognising the limitation. GPT returns to her paper.](/comp4020-ass2-jnheinrich451-eng/comics/S03-preview.webp)
Read transcript
Claude inspects changed images whose displayed feature vectors both equal [1, 2]. GPT removes her reading glasses to explain the missing information. Claude shuffles the images, then hums after recognising the limitation. GPT returns to her paper.
- Claude: The images changed, but their feature vectors are identical. Try more precision?
- GPT: More precise arithmetic cannot reconstruct information the encoder discarded.
- Claude: Eight extra decimal places. Zero extra eyesight.
Teaching labels: Same feature vectors by assumption · Same moments → same Gaussian formula · Audit the representation
Keep this: A metric inherits the distinctions its representation preserves.
This is a hypothetical exact feature match, not a claim that every visual alteration is invisible to Inception.
S04 / Week 4Choose your favourite number?Locked
Find this card's code in the optional self-check for Getting it right. You can solve the question or review its explanation.
Read transcript
GPT looks between monitors showing illustrative library scores 4.8 and 5.2, blushing as she prefers the smaller one. Claude points to resizing, weights and numerical methods. GPT scratches her head and agrees to document the implementation; Claude offers a cookie.
- GPT: Two libraries, two scores. Conveniently, I prefer the smaller one.
- Claude: Compare resizing, weights and numerical methods. Preference is not a diagnostic.
- GPT: Fine. I will pin the implementation, not the favourable result.
Teaching labels: Same images · different pipelines · Change one condition at a time · Document the chosen protocol
Keep this: Resolve the conditions of the comparison before interpreting its size.
Library scores are illustrative. Change one condition at a time; documenting an implementation supports reproducibility but does not alone establish correctness.
S05 / Week 5Same N, different baggageLocked
Find this card's code in the optional self-check for The bias. You can solve the question or review its explanation.
Read transcript
Claude interrupts GPT's bench nap with a board about equal sample sizes. GPT lifts her eye cover to inspect it. Claude thinks, ‘Generator?’, then understands the suitcase analogy. She returns to her laptop while GPT resumes her nap.
- Claude: Same N and protocol for both candidates. Their biases cancel now, right?
- GPT: Not necessarily. The finite-sample bias can depend on the generator.
- Claude: Equal-sized suitcases. Different contents.
Teaching labels: N-honest comparison · Bias(A) need not equal bias(B) · A stable ranking can still be reversed
Keep this: A fair sampling budget is necessary discipline, not a correctness guarantee.
A reversal is possible, not inevitable. Repeating the fixed-N procedure more precisely does not force its expected ordering to equal the population ordering.
S06 / Week 6A suspiciously perfect scoreLocked
Find this card's code in the optional self-check for Blind spots. You can solve the question or review its explanation.
![A suspiciously perfect score. GPT prepares to mark a task finished after seeing empirical FID(C, R) = 0. Claude says ‘Check again.’ Both displayed feature lists are [-1, 0, 0, 1]. GPT checks the copied statistics and records ‘Score audit complete’ with ‘Originality: not established’. Same moment estimators and exact arithmetic are illustrated.](/comp4020-ass2-jnheinrich451-eng/comics/S06-preview.webp)
Read transcript
GPT prepares to mark a task finished after seeing empirical FID(C, R) = 0. Claude says ‘Check again.’ Both displayed feature lists are [-1, 0, 0, 1]. GPT checks the copied statistics and records ‘Score audit complete’ with ‘Originality: not established’. Same moment estimators and exact arithmetic are illustrated.
- GPT: Zero empirical FID. My candidate must be wonderfully original.
- Claude: You copied every evaluated reference feature, with the same multiplicities.
- GPT: The score checked the statistics. It did not check my alibi.
Teaching labels: Reported distance: 0 · Identical empirical means and covariances · Zero does not certify originality
Keep this: An exact score can be correct without certifying what you wanted.
Exact arithmetic and the same moment estimators are assumed. Roundoff may leave a residual. A new independent draw or arbitrary subset does not guarantee zero.
S07 / Week 7Unbiased, not all-seeingLocked
Find this card's code in the optional self-check for Dropping the Gaussian. You can solve the question or review its explanation.
Read transcript
Claude submits a confident practice answer and receives a red cross marked Incorrect. With small tears she asks, ‘Why am I wrong?’ GPT explains the cubic kernel. Claude revises her reasoning and smiles again. The zero on the notebook is explicitly a population discrepancy; finite-sample estimates still fluctuate.
- Claude: An unbiased KID estimator! Surely candidate C has nowhere to hide.
- GPT: This kernel is cubic. C differs at degree four.
- Claude: A perfectly honest answer to a question that never asked about four.
Teaching labels: C matches R through degree three · Population discrepancy for this kernel: 0 · Estimator quality ≠ complete sensitivity
Keep this: Fixing the estimator and fixing what it measures are different tasks.
Unbiasedness concerns expectation; finite-sample estimates fluctuate. This blind spot is for the stated cubic polynomial kernel and moment-matching construction.
S08 / Week 8Wrong instrument on the witness standLocked
Find this card's code in the optional self-check for Video. You can solve the question or review its explanation.
Read transcript
GPT messages Claude from bed. Claude inspects the pooled-frame experiment at her desk and replies. GPT rolls onto her side to think, reopens the case, and sends a cartoon GPT thumbs-up sticker before putting down her phone and sleeping.
- GPT: My pooled-frame score ignores shuffling. Case closed: FVD ignores time.
- Claude: That witness is not I3D. Test the temporal change through the actual FVD pipeline.
- GPT: Case reopened. Correct instrument summoned.
Teaching labels: Experiment: pooled frame values · Different features · untested sensitivity · A toy invariance is not a universal verdict
Keep this: Transfer the question to a new instrument, not an untested conclusion.
Only the pooled-frame toy was tested. Actual FVD sensitivity remains untested here; temporal features do not guarantee sensitivity to every temporal change.
S09 / Week 9The smaller gap has a bigger billLocked
Find this card's code in the optional self-check for Human evaluation. You can solve the question or review its explanation.
Read transcript
Claude proposes halving the detectable human-error-rate gap. GPT types the calculation and shows (0.5 / 0.25)² = 4. Claude leans in, then records the increased evaluator cost while GPT sips tea. Other approximation terms and cost per judgement are held fixed.
- Claude: Let us detect half the human-error-rate gap. Half the work?
- GPT: Under the inverse-square approximation, it takes four times the judgements.
- Claude: The gap shrank. The invoice did not.
Teaching labels: 0.5 → 0.25 percentage points · (0.5 / 0.25)² = 4 · About 4× the evaluator cost
Keep this: Resolving a smaller difference can cost much more evidence.
Hold the other approximation terms, time per judgement and pay fixed. The live calculator updates variance terms and rounds counts, so its factor need not be exactly four. These are human-rating costs, not student fees.
S10 / Week 10Two uncertain rankings walk into a plotLocked
Find this card's code in the optional self-check for Correlation. You can solve the question or review its explanation.
Read transcript
Claude returns sweating from survey fieldwork, carrying human-rating sheets. From the office GPT blames the encoder. Claude replies by phone, prompting GPT to write separate metric and human protocol checklists. GPT then messages: ‘I’ve left you iced lemonade. Come back when you’re finished.’ Claude smiles at the lemon-and-ice message.
- GPT: The metric disagrees with the human ranking. The encoder is guilty.
- Claude: How uncertain is the metric ranking? And how many human trials bought the other one?
- GPT: I brought one suspect and forgot two error bars.
Teaching labels: Observed rank disagreement · Sampling on both sides · Specify both measurement protocols
Keep this: A correlation coefficient summarises a study, not just a metric.
An N-honest comparison improves interpretability but does not itself remove generator-dependent bias or make a measured human ranking exact.
S11 / Week 11Why the row staysLocked
Find this card's code in the optional self-check for Why it persists. You can solve the question or review its explanation.
Read transcript
Claude and GPT review a teaching slide. Claude proposes removing FID, then agrees to keep it with context. They add protocol and limitation notes, turn on a projector, and look at the revised slide: compatible baselines, limits of the claim, not proof of overall quality. The table contains placeholders, not measured scores.
- Claude: We found limitations. Shall we delete every FID column?
- GPT: Readers still need compatible baselines. Keep the comparison, disclose what it cannot support.
- Claude: Useful reference. Not royal decree.
Teaching labels: Known blind spots · Shared record · compatible protocols · Adoption is not a validity proof
Keep this: A useful common reference can still need qualifications.
The table contains placeholders, not measured scores. Compatible baselines do not remove finite-sample bias or certify overall quality.
S12 / Week 12The replacement takes the same examLocked
Find this card's code in the optional self-check for What you would report instead. You can solve the question or review its explanation.
Read transcript
GPT holds a new-metric proposal with a hopeful, awkward grin. Claude keeps working while listing the evidence questions. Looking caught out, GPT returns to a notebook with Checked and Missing columns. Claude notices her discouraged pause and hands her a pistachio latte. The notebook is still unfinished.
- GPT: My replacement is unbiased. I would like to skip the awkward questions.
- Claude: Blind spots? Human agreement? N-honest evidence? Cost of comparison?
- GPT: New name badge. Same evidence queue.
Teaching labels: A new metric proposal · The same standard applies · Report what was checked and what is missing
Keep this: Apply the same standard to a replacement that you applied to FID.
Compare evidence and supported conclusions, not raw magnitudes from differently scaled metrics. The optional comic is not an additional assessment.
Spoilers
This optional shortcut reveals every comic without collecting it. Press the button three times; no rapid clicking is required. Keyboard activation works too.
Review all 17 questions and explanations ↗