Week 9 Workshop: The curse of recursion

Under what conditions can recursive sampling lose rare cases?

28 April 2027

Under what conditions can recursive sampling lose rare cases?

Learning target. Compare stated conditions rather than infer training history from style.

Before the workshop#

Download the week 9 pack: a fixed six-category distribution (categories A–F, category F fixed at 3% true probability), 200 draws per round, 5 rounds after the shared round-0 draw, three seeds per condition. settings.json names every category’s true probability, the round size, the seeds, and exactly how each condition estimates its next round’s sampling distribution — everything needed to reproduce every count in this pack. Do not open reveal/ yet.

Reading. Shumailov, Shumaylov, Zhao, Papernot, Anderson and Gal, “AI models collapse when trained on recursively generated data,” Nature 631, 755–759 (2024). Access note: this article sits behind Nature’s login wall, so this course does not link a copy; the lecture states the specific claim being used — that repeatedly training on a model’s own generated output causes early error accumulation and, later, the permanent loss of low-frequency (“tail”) cases — and this workshop tests a small, fully stated instance of that mechanism rather than asking you to read the paper’s own experiments.

Check — 5 minutes#

The rare category F’s count across three seeds of the “fully recursive” condition, at rounds 3, 4 and 5: seed 11 — 5, 9, 11; seed 23 — 9, 9, 12; seed 47 — 0, 0, 0. Does this table support the claim “recursive resampling always loses rare categories by round 5”? State your answer and why in one sentence.

Receive — 15 minutes#

Confirm you can state, in your own words, what distinguishes the three conditions: the baseline never re-estimates its sampling distribution from a generated sample; the fully recursive condition estimates every later round entirely from the previous round’s own sample; the retained-data condition mixes the previous round’s sample with a fresh draw from the true distribution before estimating.

If you want to see a chain re-copy from its own last output before you run the fully recursive condition on the real distribution, try the chain simulator. Read it as an analogy only: it corrupts individual characters in ordinary text, not a six-category frequency distribution, so it cannot stand in for this week’s actual model — it can only build intuition for what re-estimating from your own last sample does over many rounds.

The resampling simulator is the closer analogy: it runs the same three conditions — baseline, fully recursive, retained-data — over an editable category distribution, so you can watch a rare category’s share drift or vanish across rounds. It is still a separate, editable illustration, not this week’s own dataset or seeded draws.

Investigate — 40 minutes#

Inspect condition definitions and sample counts (10 minutes). Read settings.json and counts.csv. Confirm round 0 is identical in method across all three conditions (a draw from the true distribution) and that the three conditions only diverge from round 1 onward.

Compare five rounds across the three conditions and three seeds (20 minutes). Using rare-category-retention.csv, plot.svg or trend.txt (accessible route), track category F’s count per condition per seed across rounds 0–5. Note which seeds, in which condition, ever reach a count of zero — and whether any run that reaches zero ever recovers.

Write a conditional conclusion and one limitation (10 minutes). State a conclusion of the form “under condition X, with these settings, category F was lost in N of 3 seeds by round 5” — not a general claim about recursive training. Then name one limitation: what property of a real trained language model (architecture, optimisation, token vocabulary, training data scale) this six-category toy has no analogue for.

Evaluate — 30 minutes#

Write your rare-category retention table (one row per condition, one column per seed, values are round-5 counts and whether zero was ever reached) and your conditional conclusion with its named limitation — before opening reveal/.

Deposit — 30 minutes#

Open reveal/analysis.md (ask your tutor for the link once your table is done) and compare it against your own conclusion. Deposit your retention table, your conditional conclusion and your limitation note into the Distribution model study, labelled a simplified model throughout.

Accessible route#

Count tables and an annotated chart description: rare-category-retention.csv and trend.txt give the same per-round, per-seed counts as plot.svg without requiring the image.

Assessment#

The Stemma is due this Friday, 2027-04-30, 12:00 Canberra time. See the assessment brief for the submission format if you have not already submitted.