Week 4: Evaluate, Decode, and Package a Base Model

Hold-out and contamination, checkpoint selection, decoding as an experimental variable, and reproducible submission packaging

15 March 2027

One frozen checkpoint branching into three decoded candidate outputs at different decoding settings, each candidate feeding into both an automatic metric box and a blind human review box.
Original illustration for SLOP4225.

Stage: Build language ability · Project 1 due

Summary

The final checkpoint is not automatically the best checkpoint, and the lowest loss does not automatically produce the best stories. This lecture treats evaluation, decoding, and packaging as parts of the training system. Students learn how document-level hold-out and contamination checks protect inference, how continuation-only PPL is calculated under the course protocol, and why tokenizer-normalised comparison is required. Temperature, top-k, top-p, repetition, EOS behaviour, and context length are examined as controlled inference choices rather than post-hoc ways to cherry-pick an attractive sample. The lecture ends with a fresh-environment loading test and a claim–evidence checklist for the report.

Learning outcomes

After this lecture, you should be able to:

  1. select a checkpoint using pre-declared quantitative and qualitative evidence;
  2. compute and interpret continuation-only PPL without leaking prompt or evaluation data into training;
  3. explain how temperature, top-k, and top-p reshape a next-token distribution;
  4. diagnose repetition, incoherence, premature stopping, and failure to emit EOS; and
  5. package a checkpoint, tokenizer, configuration, and loading command so another person can reproduce the result.

1. Evaluation separation and contamination

We revisit source-level splits and distinguish the five diagnostic examples from the ten official tutor prompts. Public prompts are not permission to train on them. Exact and near-duplicate checks must run before training, and any discovered overlap is removed or reported according to policy.

2. Checkpoints and comparable metrics

Students compare validation evidence across checkpoints generated by the same declared pipeline. Continuation-only reference-token-normalised PPL supports the course threshold even when a student changes tokenizer; bits per byte is introduced as a useful additional diagnostic. Neither metric resolves open-ended narrative quality, so output review remains necessary.

3. Decoding is an experimental variable

Greedy decoding, temperature, top-k, and nucleus sampling make different probability/variety trade-offs. A fair comparison holds prompts, seeds or seed policy, maximum length, and sampler settings constant. Students inspect how low diversity produces repetition and how excessive randomness breaks local coherence. EOS training and stopping rules are treated separately from arbitrary truncation.

4. Reproducible submission

A valid model package includes the frozen weights, architecture/ configuration, tokenizer assets, sampler assumptions, revision, checksum, and exact load/evaluate commands. The fresh-process test is compulsory engineering hygiene: success in an in-memory notebook is not evidence that the submitted artifact can be marked.

Required reading

Optional reading

This week’s action

Deadline: Project 1, Sunday 21 March 2027 at 23:59 AET. Run the final loading command in a fresh process before submission.

Teaching team