A language model is not a sentence database#

It estimates a distribution, not a lookup#

For a fixed prefix, the model outputs one probability per candidate next token — a distribution that changes as the prefix changes.

From one token to a whole sequence#

The chain rule turns local next-token predictions into a probability for a complete sequence:

P(x₁, …, x_T) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_T₋₁)

Sampling repeatedly from the changing next-token distribution — one draw, append, repeat — is how a trained model produces text.

Discuss: does “the model has a probability for this exact sentence” mean the sentence was ever seen during training?

What n-grams teach us#

The trade-off: a short context is countable but weak; a long context is informative but suffers from sparsity — most long combinations never appear in any corpus.

Where neural models change the picture#

Instead of an explicit count table per context, a neural language model learns a differentiable function from context to next-token distribution.

• shares statistical strength across similar contexts, rather than treating each exact context as a separate bucket
• can assign sensible probability to a context it has never exactly seen before, by generalising from similar ones
• is trained by gradient descent, not by counting

Takeaway: the shift from n-grams to neural models is a shift from counting exact contexts to learning a smooth function over them.

Turning a corpus into a training objective#

For a token sequence x₁, …, x_T, teacher forcing creates many supervised targets from one document: at every position, the true prefix is given, and the true next token is the label.

One document of length T yields T-1 supervised next-token problems.

Mean negative log-likelihood is cross-entropy#

Training maximises the probability the model assigns to the true next token at every position — equivalently, it minimises the mean negative log-likelihood of the observed tokens.

NLL for one token: −log P(true next token | prefix)

Averaged over all positions in a batch, this is exactly token-level cross-entropy loss, the quantity loss.backward() differentiates with respect to model parameters so an optimiser can update them.

Same objective, three names: maximum likelihood, negative log-likelihood, cross-entropy.

Perplexity: exponentiated average NLL#

Perplexity = exp(mean negative log-likelihood)

• useful for comparing checkpoints evaluated with a common token-accounting method
• roughly reads as “how many tokens the model was, on average, as unsure as choosing uniformly among”
• lower is better, but “better at predicting the next token” is not the same claim as “better at telling a coherent story”

Discuss: name one way a model could achieve very low perplexity while still producing an unsatisfying story.

Splits must respect story boundaries#

Perplexity’s evidence is only as good as the split it is measured on.

The trap: cutting token windows into train/validation/test first can put near-identical windows from the same story on both sides of the split — the model then looks good by partial memorisation, not generalisation.

The fix: create the split at the story or source level, before any token windows are formed.

The three-way split#

Decision: is this evidence enough?#

A fictional Project 1 checkpoint: validation perplexity has dropped 15% since the previous checkpoint, and the split was made at the story level.

Choose: is this alone sufficient evidence that the model tells better stories now? What additional evidence would you ask for?

Project 1’s evaluation combines both#

Project 1 does not rely on perplexity alone:

1. continuation-only, reference-token-normalised PPL — a quantitative, comparable-across-checkpoints signal
2. blind human review — a qualitative check for coherence and originality that a single number cannot carry
3. qualitative failure analysis — reading actual generated samples for specific failure patterns (repetition, incoherence, memorisation)

Takeaway: one number selects candidates; reading actual text decides between them.

Figure and source register#

• Next-token distribution diagram: original schematic drawn for this course, reusing the concept from this lecture’s own banner illustration — not a screenshot of any trained model’s actual output.
• Required reading: Ronen Eldan and Yuanzhi Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, Sections 1–2.
• Optional reading: Alec Radford et al., Improving Language Understanding by Generative Pre-Training, introduction and Figure 1.
• Course facts (teacher-forcing example, split definitions, evaluation protocol): restated from src/content/lectures/week-01.md and the Project 1 assessment page, not screenshotted, since no starter-repository run exists yet to capture.