Week 11: Administering an examination
Design a short instrument from the course's ladder, resources and notice decisions, then administer it to a real person and interpret what the record does and does not establish.
3 May 2027
Where Week 10 left us#
Week 10 showed why a performance with a collaborator cannot automatically be attributed to one person. This week turns that limit into an examination decision and retrieves Week 9’s prior-state record. We reuse Week 5’s seven rungs: Recognition, Recall, Reconstruction, Explanation, Justification, Challenge and Boundary. They classify responses under stated conditions; they do not certify understanding.
How do you actually find out what someone understands?
This is a self-guided lesson. These notes and the slide deck each contain the explanations you need; no presenter is assumed. Allow about 40 minutes for this lesson, and a separate session with another person for the Week 11 investigation, whose core task cannot be completed alone. Wren and Petra, the people in this lesson’s worked example, are fictional, distinct from Sam, Nadia, Theo, Priya, Rowan and Elin in earlier weeks.
What you will learn#
By the end you should be able to name the five decisions an examination requires before any question is asked; design a short instrument that aims specific tasks at specific rungs of Week 5’s ladder; state precisely what a solo rehearsal of this week’s activity does and does not establish, compared with administering the real thing to another person; and explain why designing a good examination is a different skill from possessing expert knowledge of its topic.
1. A different kind of task#
Every earlier week gave you a performance to read and interpret: Sam’s sentence in Week 1, Theo’s zip account in Week 2, Rowan’s three conditions on one lock in Week 4, Elin’s unannounced follow-up in Week 8. This week asks something harder. You do not read a performance someone else produced — you build the situation that produces one.
That is a genuinely separate skill from knowing the ladder. Being able to correctly classify Rowan’s condition (c) as reaching Justification and Boundary in Week 5 does not, by itself, tell you how to write a question good enough to get an answer worth classifying in the first place. This week is about that second skill.
2. Meet Wren and Petra#
Wren (fictional) wants to find out whether a friend, Petra (also fictional), genuinely understands home espresso brewing — grind size, extraction time, ratio — beyond being able to reliably make a good cup. Petra has brewed coffee daily for about two years and talks about it with confidence.
Wren’s question is this course’s question, applied for real, outside a classroom exercise: what would actually tell Wren something, rather than just make the conversation feel informative? The rest of this lesson follows Wren through the decisions that turn that question into an examination.
3. Five decisions, made before any question is asked#
Week 5 defined an instrument as “a stated rule for turning a performance into a judgement, written before you look at the case it will be applied to.” Administering an examination means making five such decisions in advance, each one reused from an earlier week and now applied on purpose rather than studied in the abstract:
- Topic and subject — someone who actually has standing to know it.
- Resources — Week 6’s question: what, if anything, may they consult?
- Notice — Week 8’s question: how much do they know is coming?
- Target rungs — Week 5’s ladder: which of the seven are you trying to produce evidence for, and with which task?
- Live collaborators — whether another person or AI may contribute during the examination, and why.
Decision 1: choose a topic and a person fairly#
Choose a real person and a real topic they already have some claim to know — not a stranger picked at random, and not a topic selected because you suspect a weakness. Wren picks Petra and coffee because Petra has genuinely practised it for two years, not to engineer a failure.
Ask first, and say roughly what you are doing and why. An examination someone did not agree to is not more honest for being a surprise; it is simply discourteous, and the discomfort of being examined without consent can distort the very performance you are trying to observe. Fairness is a design constraint here, not a separate ethical add-on.
Decision 2: resources, decided on purpose#
Week 6 asked whether needing support shows that someone does not understand, and concluded that the answer depends on what claim you are actually testing. Wren decides on no resources for the live conversation: notes, a phone, and a reference guide are all set aside, because the claim under test is Petra’s own working knowledge of brewing, produced from memory, in conversation.
A different claim — can Petra accurately interpret and apply a printed brewing guide? — would call for the opposite resources decision. Getting this wrong does not make the examination harder or easier; it makes it a test of something other than what you intended, exactly the trap Week 6 warned about.
Decision 3: notice, decided on purpose#
Week 8 distinguished a rehearsed performance from an unannounced one, and found that surprise can reveal structure that a fully prepared answer might have hidden. Wren tells Petra in advance: “I want to talk through how you brew coffee, and I will probably ask some follow-up questions.”
That is partial notice. Petra knows the topic and that follow-ups are coming, but not their content or order. Full advance notice of every specific question would let Petra rehearse polished answers for each one; giving no notice at all would tip from an examination into something closer to an ambush, and would also raise the fairness problem from Decision 1 again. Partial notice is a deliberate middle position, chosen for what it can reveal without being unkind.
Decision 4: aim tasks at specific rungs#
A single question tends to produce evidence for a single rung of Week 5’s ladder, so a short instrument needs several, each aimed on purpose rather than picked at random. Wren writes four tasks in advance:
| Task | Question | Rung aimed at |
|---|---|---|
| 1 | “What water-to-coffee ratio do you use?” | Recall |
| 2 | “Walk me through your process from grinding to pouring, without looking at anything.” | Reconstruction |
| 3 | “Why does a finer grind need a shorter extraction time?” | Justification |
| 4 | “When has your usual method actually failed you?” | Boundary |
Writing the target rung down before asking the question is what makes this an instrument rather than a conversation that happens to touch on coffee. It also lets Wren notice, afterward, whether an answer landed on a different rung than the one the task was aimed at — section 6 shows exactly that happening.
Collaborator decision and prior-state record#
Decision 5: live collaborators. State whether another person or AI may contribute while the subject answers, and why. Wren excludes live collaborators to examine Petra’s own responses; an assisted version would require a record of who contributed what and a claim about the joint performance.
Before the first task, ask the subject to record one claim about the topic, a reason and their confidence, before consulting anything or receiving feedback. Keep it unchanged beside the final answer. This Week 9 prior state establishes what was recorded at that point, not everything they previously knew.
For example, a fictional prior note might say: “A finer grind needs less extraction time because water contacts more surface; confidence: medium.” Later agreement shows that the claim was recorded before the tasks; it does not prove that every later justification was already available.
4. Pause and design your own · 6 minutes#
Pick a real person you could plausibly examine, and a topic they have genuine standing to know — a hobby, a job skill, something they have actually practised, not a course topic they would have to fake competence in.
Write down your five decisions: the topic and person; your resources decision and the one-sentence reason for it; your collaborator decision and reason; your notice decision and the one-sentence reason for it; and three tasks, each labelled with the rung from Week 5 it is aimed at. Attempt this before reading the worked response below.
Worked response: reading your own design
There is no single correct instrument, but a defensible one has a stated reason attached to every decision, not only a chosen topic. Check your draft: did you write down why you set the resources condition the way you did, or only that you set it? The same check applies to notice.
If two or three of your tasks are aimed at the same rung, that is not a mistake, but it is worth noticing — it likely means one whole kind of evidence (Challenge, say, or Boundary) will never get produced by this instrument, the same gap Week 5 found across four weeks of solo performances by Sam, Theo, Priya and Rowan.
5. Design is solo-completable; administration is not#
This is the one point in the course where the guideline is explicit rather than implicit: a solo rehearsal of this week’s activity does not fulfil its separate aim. You can design an instrument alone, on paper, and that is a genuinely useful exercise for the skill in sections 3 and 4. But administering it requires a second, independent mind producing answers you did not write and cannot fully predict — that independence is exactly what makes the resulting evidence worth anything at all.
If you rehearse this lesson’s exercise by asking yourself the four questions, you will learn something about instrument design, but you will not produce examination evidence, because you already know every answer you are about to give. The Week 11 investigation states this distinction again, explicitly, before its instructions.
6. Worked administration: Wren and Petra talk#
Wren asks the four questions in order, without notes of her own beyond the prepared task list. Petra gives the ratio correctly — Recall. She reconstructs her full process fluently, in order, including grind setting, bloom time, and pour pattern, with nothing to consult — Reconstruction.
Asked why a finer grind needs a shorter extraction time, Petra says: “Because more surface area means the water pulls flavor out faster — leave it too long and it goes bitter.” That is Justification: a reason for the claim, not merely a restated rule, matching Week 5’s definition of the rung exactly.
Asked the fourth, Boundary-aimed question — when has her usual method actually failed her — Petra pauses, then answers: “Humid days. I’ve never worked out why, I just add a bit more coffee and it comes out right, but I couldn’t tell you the mechanism.” Wren had not anticipated this specific answer. It is genuine Boundary-rung evidence: Petra names, unprompted, a condition under which her own account runs out, exactly Week 5’s definition of the rung. It also reveals something Wren’s instrument design could not have predicted in advance — that Petra’s Boundary answer names an unexplained mechanism, while her Justification answer supplied one cleanly, an unevenness across rungs the task list alone did not surface until the conversation happened.
7. Interpreting the record#
Wren now holds four data points spanning four different rungs, produced live, from memory, under a notice condition decided on purpose. Compare this with Week 1: one sentence from Sam, read from an open reference, is a much thinner record than what Wren now has from Petra. Compare it with Week 4: three conditions on one unaided person is closer in scale, but Rowan’s task there was fixed in advance by the lesson, not designed and administered by a peer in conversation.
It is still not certain. Petra could plausibly have explained the extraction-time relationship to other people, in nearly these words, many times before this conversation; Wren has no way to rule that out from a single sitting. A stronger record would need either a topic where rehearsal is less likely, or a second, differently structured conversation later, to see whether the same reasoning reappears in a different form.
8. What the instrument still cannot detect#
However carefully designed, Wren’s instrument cannot establish whether Petra’s Justification answer existed, worded roughly this way, before the question was asked, or was assembled fluently in the moment of being asked — the exact limit Week 8 found with Elin’s unannounced follow-up about salt and fermentation. Nor can it distinguish genuine, connected understanding from a well-rehearsed but shallow explanation, if Petra happens to have one for this particular topic.
An instrument narrows what a performance could mean; four separate answers, each aimed at a different rung, rule out more alternative explanations than one lucky sentence would. It does not settle what the performance does mean. That gap between narrowing and settling is precisely what Week 12 takes up next.
9. Knowing a topic is not the same skill as examining it#
Wren does not brew espresso to a professional standard, and did not need to, in order to design and run this examination well. What Wren needed was Week 5’s ladder, a resources decision made for a stated reason, a notice decision made for a stated reason, and the discipline to write four tasks, each aimed at a specific rung, before hearing a single answer.
This is worth stating directly, because the opposite assumption is easy to make by default: a good examiner does not need to out-perform the person being examined on the topic itself. They need a good instrument, built from decisions like the five above, and the judgement to interpret honestly what that instrument returns — including, as section 6 showed, the parts it did not predict.
10. Fairness is not a separate concern from good design#
A technically well-designed examination can still be unfair: sprung on someone without consent, aimed at a topic they never claimed to know, or scored against a rung the task never actually gave them room to reach. Recall Week 5 section 4: Challenge cannot be produced by a person working alone, because it requires a second party to supply the attack — so an instrument that never gives its subject a genuine objection to respond to cannot fault them for failing to reach that rung.
Wren’s design passes this check on all three counts: Petra agreed to the conversation and knew roughly what it was for; the topic is ground Petra genuinely stands on; and every rung Wren scored in section 6 had a task in the original list built specifically to reach it, so no verdict was reached by a question that was never actually asked.
11. Keep a short record#
Before you finish designing, or immediately after you administer your instrument, write five lines and keep them.
| Line | What to write |
|---|---|
| Prior state and collaborators | The unchanged initial claim, reason and confidence; permitted and actual collaborator contributions. |
| Resources and notice | What you decided for each, and the one-sentence reason it matched the claim you wanted to test. |
| Rungs reached | Which of the seven rungs your subject’s performance reached, quoting the specific words that earned each one. |
| Surprise | Did any answer land on a rung you had not aimed a task at, the way Petra’s Boundary answer did for Wren? |
| Remaining uncertainty | What this record still cannot rule out, stated as precisely as section 8’s limits above. |
Reading with a purpose#
Synthesis text on argument structure. Booth, W. C., Colomb, G. G., and Williams, J. M. (2016). The Craft of Research, 4th ed. University of Chicago Press. Chapter 8, “Making Good Arguments: An Overview,” sets out how a claim needs stated reasons, evidence, and acknowledged objections to be defensible — the discipline behind writing down why each of this week’s five decisions was made, not only what was decided. Ask: which of your five decisions in section 4 has the weakest stated reason, and what would strengthen it? This is used here as a tool for examining an instrument’s design, not as a claim about cognition, coffee, or examinations generally — the book makes no such claim.
What this leaves unresolved#
In this fictional example, Wren now has a record: four rungs, reached live, by another person, under resources and notice decisions made in advance and stated reasons attached to each. It is the strongest evidence this course has produced so far — and section 8 already showed it still ends in a judgement Wren has to make, not a number the instrument hands over automatically.
Apply this yourself in the Week 11 investigation, which asks you to design an instrument on paper first, then actually administer it to a real person and keep the record described in section 11. Week 12 takes a record exactly like the one Wren now holds and asks the final question of the course: what are you actually justified in concluding from evidence like this — and where, precisely, does that justification run out?