Administering an examination#
You’ve built a ladder, tested resources, changed form, and been surprised. Now: how do you actually run an examination, start to finish?
Week 11 · A self-guided lesson
Where Week 10 left us#
Week 10 showed why a performance with a collaborator cannot automatically be attributed to one person. This week turns that limit into an examination decision and retrieves Week 9’s prior-state record. We reuse Week 5’s seven rungs: Recognition, Recall, Reconstruction, Explanation, Justification, Challenge and Boundary. They classify responses under stated conditions; they do not certify understanding.
How to use this lesson#
Read at your own pace. Allow around 40 minutes for the lesson, and a further session with another person for the full investigation — this week’s core task cannot be completed alone.
You will design a short instrument here, using ideas from Weeks 5 to 9. The Week 11 investigation then asks you to run it on someone real.
A different kind of task#
Every earlier week gave you a performance to read — Sam’s sentence, Theo’s zip, Rowan’s lock, Elin’s dough — and asked what it showed. This week asks something harder: you build the situation that produces the performance.
That is a distinct skill from knowing the ladder. Being able to classify Rowan’s answer in Week 5 does not, by itself, tell you how to design a question good enough to get an answer worth classifying.
Meet Wren#
Wren (fictional) wants to find out whether a friend, Petra (also fictional), genuinely understands home espresso brewing — grind size, extraction time, ratio — beyond being able to make a good cup.
Petra has made coffee daily for two years and talks about it confidently. Wren’s question is the course’s question, applied for real: what would actually tell us, rather than just make us feel informed?
Wren and Petra are new fictional characters, distinct from Sam, Nadia, Theo, Priya, Rowan and Elin.
Five decisions, before any question is asked#
Designing an instrument means making decisions in advance, the way Week 5 defined an instrument as “a stated rule… written before you look.”
- Topic and subject — something they actually have standing to know.
- Resources — Week 6’s question, decided on purpose this time.
- Notice — Week 8’s question: how much do they know is coming?
- What rungs you’re aiming at — Week 5’s ladder, chosen deliberately.
- Live collaborators — whether another person or AI may contribute during the examination, and why.
Decision 1: choose a topic and a person fairly#
Choose a real person and a real topic they already have some claim to know — not a stranger, not a subject picked to catch them out. Wren picks Petra and coffee because Petra has genuinely practised it, not because Wren suspects a weakness.
Ask first, and say roughly what you’re doing. An examination someone didn’t agree to isn’t more honest for being a surprise — it’s just discourteous, and the discomfort can distort the performance you get.
Decision 2: resources — Week 6’s question, decided on purpose#
Week 6 asked whether needing support shows you don’t understand. Here, you choose deliberately: will notes, a phone, or a reference be available?
Wren decides on no resources for the live conversation — the claim under test is Petra’s own working knowledge of brewing, produced from memory, not her ability to look something up. A different claim (can Petra interpret a brewing guide accurately?) would call for the opposite decision.
Decision 3: notice — Week 8’s question, decided on purpose#
Week 8 distinguished rehearsed from unannounced performance. Wren tells Petra in advance: “I want to talk through how you brew coffee, and I’ll probably ask some follow-up questions.”
That is partial notice: Petra knows the topic and that follow-ups are coming, but not their content. Full advance notice of every question would let Petra rehearse; zero notice at all would be closer to an ambush than an examination.
Decision 4: aim at more than one rung#
A single question tends to produce evidence for a single rung. Wren writes four tasks in advance, each aimed at a different one:
Aimed low
Recall: “What water-to-coffee ratio do you use?” Reconstruction: “Walk me through your process from grinding to pouring, without looking at anything.”Aimed high
Justification: “Why does a finer grind need a shorter extraction time?” Boundary: “When has your usual method actually failed you?”Decision 5 and the prior state#
Decision 5: live collaborators. State whether another person or AI may contribute while the subject answers, and why. Wren excludes live collaborators to examine Petra’s own responses; an assisted version would require a record of who contributed what and a claim about the joint performance.
Preserve a prior state#
Before the first task, ask the subject to record one claim about the topic, a reason and their confidence, before consulting anything or receiving feedback. Keep it unchanged beside the final answer. This Week 9 prior state establishes what was recorded at that point, not everything they previously knew.
For example, a fictional prior note might say: “A finer grind needs less extraction time because water contacts more surface; confidence: medium.” Later agreement shows that the claim was recorded before the tasks; it does not prove that every later justification was already available.#
Pause and write · 6 minutes#
Pick a real person you could plausibly examine, and a topic they have genuine standing to know (a hobby, a job skill, something they’ve actually practised — not a course topic they’d have to fake).
Write down your five decisions: the topic and person; your resources decision and why; your collaborator decision and why; your notice decision and why; and three tasks, each labelled with the rung from Week 5 it’s aimed at. Attempt this before reading on.
Worked response: reading your own design#
There’s no single correct instrument, but a defensible one has reasons attached to every decision, not just a topic. Check your draft: did you write down why you chose that resources setting, or just that you chose one?
If two of your three tasks are aimed at the same rung, that’s not wrong, but notice it — it means one whole kind of evidence (say, Challenge, or Boundary) may never get produced, the same gap Week 5 found across four weeks of solo performances.
Now, the part you cannot do alone#
Design is solo-completable. Administering it is not. You need a second, independent mind producing a performance you did not write and cannot predict in full — that is what makes the resulting evidence worth anything.
If you rehearse this lesson’s exercise on yourself, you can still learn the design skill, but you will not produce examination evidence, only a design sketch. The Week 11 investigation states exactly what a solo attempt loses.
Worked administration: Wren and Petra talk#
Wren asks the four questions in order. Petra gives the ratio correctly (Recall). She reconstructs her process fluently, including grind, bloom, and pour timing (Reconstruction). Asked why fine grind needs shorter extraction, she says: “Because more surface area means the water pulls flavor out faster — too long and it goes bitter.” That is Justification: a reason, not just a rule.
The follow-up Wren didn’t plan for#
Asked about the Boundary question — when has her method failed — Petra pauses, then says: “Humid days. I’ve never worked out why, I just add a bit more coffee and it comes out right, but I couldn’t tell you the mechanism.”
Wren did not expect this. It is genuine Boundary-rung evidence — Petra names, unprompted, where her own account runs out — but it also reveals that her Justification rung was stronger than her Boundary rung, which Wren could not have predicted from the ratio alone.
Interpreting the record#
Wren now has four data points across four rungs, produced live, with no notes, under partial notice. That is stronger evidence than Week 1’s single sentence from Sam, or Week 4’s three conditions on one unaided person — it spans more of the ladder, from one real conversation.
It is still not certain. Petra could have rehearsed this exact ratio-and-process explanation many times before for other people; Wren has no way to rule that out from this conversation alone.
What the instrument still cannot detect#
However well-designed, this instrument cannot tell Wren whether Petra’s Justification answer existed before the question, or was assembled fluently in the moment — the same limit Week 8 found with Elin’s follow-up. It also cannot separate genuine understanding from a well-rehearsed but shallow script, if Petra happens to have one.
An instrument narrows what a performance could mean. It does not settle what it does mean. That gap is Week 12’s subject, not this week’s.
Knowing a topic is not the same skill as examining it#
Wren does not brew coffee expertly, and did not need to. What Wren needed was the ladder, a resources decision, a notice decision, and the discipline to write tasks before hearing any answers.
This is worth stating directly, because it is easy to assume otherwise: a good examiner does not need to out-perform the person being examined. They need a good instrument and the judgement to interpret what it returns.
Fairness is not an afterthought#
An examination can be well-designed and still unfair: sprung without consent, aimed at a topic the person never claimed to know, or scored against a rung the task never gave them room to reach (recall Week 5 section 4 — Challenge needs a second party; you cannot fault someone for not producing it alone).
Wren’s design passes this check: Petra agreed, chose ground she actually stands on, and every rung Wren scored had a task built to reach it.
Keep a short record#
Before you finish designing (or after you administer), write five lines and keep them:
| Line | What to write |
|---|---|
| Prior state and collaborators | The unchanged initial claim, reason and confidence; permitted and actual collaborator contributions. |
| Resources and notice | What you decided, and why that decision matched the claim you wanted to test. |
| Rungs reached | Which of the seven your subject’s performance reached, with the specific words that earned each one. |
| Surprise | Did any answer reach a rung you hadn’t aimed a task at? |
| Remaining uncertainty | What this record still cannot rule out. |
What this hands to Week 12#
In this fictional example, Wren now has a record: four rungs, reached live, by another person, under decisions made in advance. It is the strongest evidence this course has produced so far — and it still ends in a judgement Wren has to make, not a number the instrument hands over automatically.
Final problem: what are you actually justified in concluding from evidence like this? Week 12 asks you to interpret your own record alongside a new fictional hard case and asks what it can, and cannot, support.
Read further: interpreting evidence#
Booth, W. C., Colomb, G. G., and Williams, J. M. (2016). The Craft of Research, 4th ed. University of Chicago Press. Chapter 8 (Making Good Arguments: An Overview) sets out how a claim needs reasons, evidence, and acknowledgment of objections to be defensible — the same discipline Wren’s decision log applies to an examination design. Ask: which of your five decisions in this lesson has the weakest stated reason?
Sources and your next step#
Booth, Colomb & Williams (2016), The Craft of Research, 4th ed., chapter 8. This is a synthesis text on argument structure, used here to ground why an instrument’s decisions need reasons, not as evidence about cognition or coffee.
Wren, Petra, and their conversation are fictional teaching material, not a reported case.
For the full instrument-design-and-administration activity, open the Week 11 investigation. The lecture notes give a continuous reading version with the same worked example in more depth.