Scale is an allocation problem#
With one GPU-day, would you spend it on a larger model, more distinct tokens, or more updates over the same data?
Project 1 has three coupled quantities#
- N — learned parameters
- D — training tokens processed
- C — training compute
Project 1’s constraints, in one table#
| Constraint | Value |
|---|---|
| Design target | 32M learned parameters |
| Absolute eligibility boundary | 33.6M learned parameters (5% tolerance) |
| Compute allowance | 24 T4-equivalent hours |
| Training FLOP ceiling | 4.0×10¹⁷ |
Takeaway: every proposed run must fit all of these at once — a plan that saves parameters but blows the FLOP ceiling is not a legal plan.
Bigger is not automatically better#
Three ways a plan can fail without touching the parameter boundary:
- too many parameters for the data on hand — capacity the corpus can’t fill
- many repeated passes over a narrow corpus — compute spent, little new information gained
- a tiny model trained long after useful validation improvement stopped — time spent past the point of return
Discuss: for each sketch, which resource — model, data, or compute — is actually the bottleneck?
Power-law intuition#
The power law, written out#
L(x) = a·x⁻ᵅ + L∞ — diminishing returns, and an approximately straight middle region on log–log axes.
Kaplan et al.: smooth scale relationships#
- reported smooth relationships between loss and N, D, C across many orders of magnitude
- larger models are more sample-efficient — they reach a given loss with fewer tokens
- weak sensitivity to architectural shape within the studied range
- early stopping is compute-optimal: stop before full convergence on a fixed budget
Callout: the studied regimes are far above 32M. Read this as evidence of a shape, not a recipe for this course’s model.
Chinchilla changed the allocation#
| Question | Earlier view (Kaplan-era) | Chinchilla-era view |
|---|---|---|
| Under fixed compute, favour… | more parameters | parameters and tokens roughly together |
| Main risk | — | undertrained large models |
The iso-compute valley#
Takeaway: an undertrained large model can lose to a smaller model trained on more tokens, under the same compute.
What transfers to 32M?#
| Transferable | Not guaranteed at this scale |
|---|---|
| Identify which resource is the bottleneck | The published exponents |
| Compare configurations within a fixed budget | An optimal token/parameter ratio |
| Allocate data with model size | A specific batch size |
| Stop by evidence, not by habit | An exact loss value for this corpus |
Decision 1: choose a plan#
Three legal fictional plans, same maximum compute:
| Plan | Parameters | Tokens |
|---|---|---|
| A | 18M | more tokens |
| B | 31.8M | balanced tokens |
| C | 33.5M | fewer tokens |
Choose: not a winner — which uncertainty would you need to measure before picking one, given only this table?
A token is not a unit of value#
| Source | Quality signal |
|---|---|
| Clean, diverse stories | high value per token |
| Duplicated stories | diminishing value with repetition |
| OCR noise | actively harmful |
| Synthetic textbook-like stories | can raise value per token, not guaranteed |
Quality, diversity, duplication, provenance, and domain fit all change what a token is worth.
Count data twice#
Two ledgers, not one:
- unique source/token inventory — how much distinct material exists
- tokens processed during optimisation — how much the optimiser actually saw, including repeats
Multiple epochs increase the second without increasing the first — a report needs both numbers, or a duplication claim can’t be checked.
From batch to compute#
- sequence length × sequences per micro-batch × gradient accumulation → effective tokens per update
- updates × effective tokens per update → total tokens processed
The repository is canonical for hardware conversion and FLOP estimation — the ledger relationship above is for reasoning about a plan, not for computing a number to report without running the tool.
Learning rate is a schedule, not one number#
Early instability calls for a short warm-up; later refinement calls for decay. An aggressive rate late in training can destroy a useful checkpoint — a lesson that carries forward directly into Projects 2 and 3’s post-training stages.
Stability signals#
| Signal | Warning pattern | Ambiguous pattern |
|---|---|---|
| Train/validation loss | validation rises while train falls | validation flattens early |
| Gradient norm | sudden spike or divergence | one isolated spike |
| Throughput/memory | steadily degrading throughput | one slow step |
| Fixed-prompt samples | repeated or degenerate text | one attractive sample |
A single spike, or one attractive sample, is not a conclusion on its own.
A pilot answers one uncertainty#
Template: hypothesis → one changed factor → controlled variables → maximum spend → measurement window → stop rule → decision the result can change.
Contrast this with an undirected hyperparameter sweep, which spends budget without naming what it will decide.
Decision 2: continue, revise, or stop?#
A fictional pilot: training loss falls, validation loss flattens, gradient norm is stable, projected runtime is 30 hours, samples repeat source phrases.
Choose an action, and name the evidence for it.
Your Project 1 run plan#
Final checklist before spending the principal run:
- legal preflight passed
- clean, source-level split
- unique and processed token counts recorded
- baseline plus one controlled comparison
Your Project 1 run plan, continued#
- compute reservation held back
- checkpoint interval set
- fixed evaluation prompts ready
- a stop rule and recovery margin
Next: bring this checklist to the Week 3 drop-in clinic.
Claims you may and may not make#
| Supported | Unsupported |
|---|---|
| “Under our fixed data and steps, A achieved lower validation loss than B.” | “Width is generally better than depth.” |
| “Chinchilla proves our exact configuration is optimal.” | |
| “Lower PPL guarantees better stories.” |
Exit ticket#
Submit: your chosen baseline · the largest unresolved uncertainty · the pilot that can resolve it · maximum hours/FLOPs reserved · the evidence that would trigger a change.
Project 1 due 21 March 2027, 23:59 AET.
Figure and source register#
- Power-law and scaling claims: Kaplan et al., Scaling Laws for Neural Language Models.
- Compute-optimal allocation and iso-compute concept: Hoffmann et al., Training Compute-Optimal Large Language Models.
- Small-model/data-quality case studies: Eldan and Li, TinyStories; Gunasekar et al., Textbooks Are All You Need.
Figure and source register, continued#
- Course budget values and ledger definitions:
src/course-config.tsand the Project 1 assessment page, restated here rather than screenshotted, since no starter-repository run exists yet to capture. - The power-law, iso-compute-valley, and learning-rate-schedule diagrams in this deck are original schematics drawn for this course, illustrating the qualitative shape of the cited papers’ reported relationships — not reproductions of those papers’ own figures.