Week 12: Defending the inference

Watch a fictional examinee pass every rung of the course's own instrument cleanly, then find out why that clean pass still cannot settle what it was built to settle.

10 May 2027

Open the slides

What makes the inference defensible?#

Validity concerns how well the evidence supports the particular interpretation and use we propose. A successful performance does not validate every claim about the person. This returns to Week 1’s distinction between an observation and an inference. Farhan is a new fictional hard case; compare it with your own Week 11 record, using the supplied fallback if needed.

What can we responsibly claim from the evidence we collected?

This is a self-guided lesson, and the course’s last one. These notes and the slide deck each contain the explanations you need; no presenter is assumed. Allow about 40 minutes, with another 20 if you complete the Week 12 investigation. You need nothing open from earlier weeks, but this lesson reuses the seven-rung ladder built in Week 5 without redefining it from scratch, so it helps to have read that week once. Farhan is a new fictional examinee, distinct from Sam, Nadia, Theo, Priya, Rowan and Elin.

This lesson does not add an eighth rung to the ladder, and it does not resolve its own hard case cleanly by the end. That is the intended ending of the course, not an unfinished one.

What you will learn#

By the end you should be able to state why a performance that passes every rung of an instrument still does not settle what produced it; distinguish useful evidence from sufficient evidence for a specific claim; name a false positive and a false negative the same instrument can produce; explain why adding another rung does not fix the underlying limit; and write a claim about your own evidence that states exactly what it supports and what it does not.

1. Where the course has got to#

Weeks 1 to 4 built reasons to distrust an answer taken at face value: an observation is not an inference, a fluent explanation can conceal missing structure, a reconstruction can still be rehearsed, and two careful readers judging the same texts under different conditions can reach opposite verdicts with no way to show each other where they disagree. Week 4 closed by naming what was missing — not more evidence, but a stated rule for weighing evidence that is already in hand.

Week 5 built that rule: the seven-rung ladder, defined by what a performance has to do rather than by how confident or fluent it sounds. Recognition, Recall, Reconstruction, Explanation, Justification, Challenge, Boundary — each rung is an operational definition, quoted in full in Week 5, that lets two readers point at a specific rung and argue about it instead of trading impressions. Weeks 6 to 11 spent six weeks testing that instrument against harder conditions: what someone is permitted to consult, a changed form of the same idea, an unannounced follow-up, a recorded prior state, a collaborator’s participation, and a real administered exam. The ladder survived all of it. This week asks what surviving all of it actually proves.

2. Meet Farhan, and the topic#

Farhan is a fictional examinee. You are his examiner, following the administration standard this course has been arguing for since Week 6 and Week 11: a real exam, on a declared topic, under stated conditions, with a real live follow-up rather than a scripted one.

The topic is why the sky is blue during the day, and red or orange at sunset. Sunlight reaching the atmosphere is scattered in all directions by the gases in the air. Blue light, travelling as a shorter wave, scatters more than the longer-waved colours, so during the day scattered blue light reaches your eyes from all across the sky rather than only from the sun’s direction. Near sunset, sunlight has to travel through far more atmosphere to reach an observer; even more of the blue gets scattered away over that longer path, leaving mostly the reds and yellows to arrive directly.

Technical background: NASA Space Place, “Why is the sky blue?”. This grounds the worked example; it is not evidence about understanding or cognition, and the claim this lesson makes is no broader than what that page states.

Farhan is told the topic in advance — “something about the sky’s colour” — but not the exact questions. This is the notice distinction Week 8 introduced: full notice of the topic, no notice of the specific follow-up. Notes are removed before the exam begins, and no device is visible in the room. Nothing about the administration is careless. That will matter later.

3. Farhan’s performance, rung by rung#

With the explanation briefly shown to him, Farhan repeats it accurately — Recognition. A few minutes later, with the notes removed, he retrieves the same content unprompted: “Blue light scatters more because it’s a shorter wave, and that happens in all directions, all over the sky.” That is Recall: produced from memory, nothing to consult.

Asked to rebuild the whole mechanism without being shown it again, Farhan does: “Sunlight hits the air, the gas molecules scatter the short blue wavelengths much more than the long red ones, and that scattered blue is what reaches you from every direction — that’s why the sky looks blue instead of the sun looking blue and everything else looking normal.” The parts are named, related to one another, and the causal step is stated: Reconstruction. Asked to explain the same idea to someone with no background, he recasts it with an analogy about light bouncing off small obstacles far more often than it travels straight through — pitched for a listener rather than only for himself. That is Explanation, a different demand from Reconstruction because it has to satisfy someone else.

Pressed on why shorter wavelengths would scatter more than longer ones, Farhan answers: “The explanation I read said it’s because they’re shorter, smaller waves — I don’t know the physics much deeper than that, but that’s the stated reason, not just a fact I’m repeating.” This is a defensible Justification: he gives the reason inside the explanation rather than only asserting the outcome, while also being honest about the limited depth of his own grounding — a detail this lesson returns to.

4. Rung 6: an objection nobody scripted#

You put one objection to Farhan live, unrehearsed on your side as well as his: “Isn’t that a contradiction? The daytime story says blue light reaches you because it scatters. The sunset story says blue light is lost because it scatters. Which is it?”

Nobody scripted this objection for Farhan in advance, and his response meets Week 5’s Challenge definition on its own terms: he defends the original account by showing why the objection misses, rather than revising it or dodging it. This is genuinely rare evidence — Week 5 named Challenge as the one rung nobody in the course’s first four weeks had ever produced, precisely because it needs a second party who has not already agreed with you.

5. Rung 7: a limit, volunteered#

Without being asked, Farhan adds one more sentence: “I should say — I don’t actually know why sunsets are sometimes more orange than red, or whether smoke or dust changes it. I’m confident in the scattering explanation itself, but not in the finer detail past that.”

Nobody asked him to name a limit. He found the edge of his own account and said so unprompted — exactly Week 5’s definition of Boundary. By the ladder’s own rules, Farhan has now passed all seven rungs: Recognition, Recall, Reconstruction, Explanation, Justification, Challenge, Boundary, live, under real questioning, from a real second person, with nothing to consult. On paper this is the strongest single performance the course has examined.

6. Pause and write: your verdict · 4 minutes#

Before reading further, write your own answer: does a clean pass on all seven rungs, administered this carefully, justify concluding that Farhan understands why the sky is blue and why sunsets are red? Write two or three sentences, and name the one piece of evidence your verdict depends on most.

Attempt this before continuing. The next section does not resolve it for you.

7. What the examiner learns afterward#

A friend of Farhan’s mentions, in passing, that the evening before the exam Farhan spent over an hour in conversation with an AI assistant. Told only that “something about light and colour” would be on the exam, he worked through the daytime and sunset explanations, rehearsed a response to “isn’t that a contradiction,” and was told that if pressed on the finer detail, it was fine to say plainly that he didn’t know it.

Nothing about the exam’s administration was at fault here — no device was visible, no notes were present, the follow-up was genuinely unscripted on your side. This new information arrives after the fact, second-hand, from someone who was not in the room, and is itself imperfect evidence. It does not prove that Farhan’s live answers were recited rather than reasoned. It does mean you can no longer rule that out.

8. Why this reopens the case rather than closing it#

Week 5 named an assumption sitting inside the ladder’s own wording: Recall is defined as producing material “without the source in front of me,” and Reconstruction as rebuilding relationships “without the source in view.” Both definitions check for a source being absent at the moment of the performance. Neither asks where the content itself came from before that moment.

A well-rehearsed answer, built collaboratively with an AI the night before and retrieved from memory during the exam, satisfies “produced without the source in front of me” exactly as well as an answer Farhan worked out entirely alone. The two histories are not distinguishable from the transcript, because the ladder was never built to distinguish them — it was built to tell a live performance apart from one produced while reading a source in the room, and Farhan’s device was never in the room. A rehearsed script recalled from memory is not the situation any of the seven rungs were written to catch.

9. Two verdicts, both defensible, neither final#

Verdict A: count the pass. The exam was administered correctly, on the standard the course has argued for since Week 11. Some preparation before a declared exam is normal and expected — you cannot penalise Farhan simply for having studied the announced topic the night before. Record the pass, with the preparation condition noted honestly rather than omitted.

Verdict B: mark it inconclusive. The specific objection Farhan handled so cleanly may be one he had already rehearsed an answer to, which would make it far weaker evidence of live reasoning than it appeared to be at the time. Without a topic, or an objection, that Farhan genuinely could not have anticipated, this particular exam cannot rule that possibility out.

Neither verdict is obviously the mistaken one. Holding both, and being explicit about which piece of evidence would move you from one to the other, is itself the disciplined move this lesson is asking you to practise.

10. Useful evidence is not the same as sufficient evidence#

Farhan’s performance is useful evidence of something real: under live questioning, from a real examiner, with nothing to consult, he produced a structurally complete and accurate account of this topic and responded coherently to a genuine, unscripted challenge. That is not nothing, and Verdict A is not a naive reading for treating it as real evidence of capability.

It is not sufficient evidence that the understanding behind it is independently Farhan’s, because the identical transcript is equally consistent with a well-prepared recollection of material substantially built with a collaborator. The instrument cannot tell, from the transcript alone, which explanation is correct — and no amount of additional polish in the performance closes that gap, because polish is exactly what a good rehearsal also produces.

This distinction — useful versus sufficient — is the sharpest single idea this course can leave you with. An instrument can supply real, usable evidence for a specific, narrow claim while still falling short of a much larger claim the same evidence is often mistaken for supporting.

11. Two ways this could mislead you#

A false positive: concluding “Farhan understands” from a performance that was substantially a rehearsed, AI-co-produced script, recalled convincingly under the pressure of a real exam. This is the failure mode this lesson has walked through in detail.

A false negative is worth naming as its flip side, so the lesson does not read as one-sided suspicion of every strong performance: someone who understands the material deeply but freezes, rushes, or explains it badly under live questioning could score lower on this same exam than Farhan did. An instrument built to catch rehearsed fluency can just as easily undersell real understanding that had every chance to show itself and didn’t, on the day, in that room.

12. Why one more rung will not fix this#

It is tempting to propose an eighth rung: “explain something you could not possibly have rehearsed.” This moves the problem rather than solving it. However carefully a genuinely novel topic is chosen, the examiner is still trusting that no collaborator helped Farhan with it either — live, off camera, or through preparation nobody in the room has any way to inspect.

Every rung this course has built converts an observed performance into a claim about what produced it. No rung, and no achievable number of additional rungs, can make that conversion certain, because the thing being inferred — what Farhan actually understands, independent of any collaborator — is never the thing directly observed. What is observed is always a performance; what is claimed is always more than the performance strictly shows. That gap is not a flaw in this particular ladder. It is the condition any instrument like it works under.

13. Pause and write: defend your own inference · 6 minutes#

Take a real piece of evidence from earlier in this course: your own Week 11 examination, a Week 5 or Week 9 record, or any performance you produced and kept. Write one sentence claiming what it justifies, and a second sentence stating what it does not — matching Verdict A and Verdict B’s shape above, applied to your own case rather than Farhan’s.

14. What this course leaves you with#

Twelve weeks have not produced an instrument that turns a performance into a certainty about the person who produced it. They have produced something narrower, and more genuinely useful: a way to state exactly what a piece of evidence supports, what it does not, and why — so that a conclusion can be checked, argued with by someone else pointing at the same rung, and revised when better evidence arrives, instead of simply believed because it sounded convincing.

That is what defending an inference means in this course’s sense. Not proving understanding. Defending a specific, bounded claim about what you actually observed, and being honest about the gap between that claim and the larger one it is often mistaken for.

Reading with a purpose#

Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., and FitzGerald, W. T. The Craft of Research (University of Chicago Press). Part 3, “Making an Argument,” covers claims, reasons and evidence, with dedicated sections on warrants and on acknowledging and responding to objections. This lesson uses the book at that general, structural level — a claim needs a stated warrant and an honest qualification of what it does and does not establish — which is exactly the shape of Verdict A and Verdict B above. This citation is verified at the level of the book’s public table of contents, not from a page-level reading of the text; no page number or direct quotation is claimed here, and none should be added without reading the relevant pages directly.

Rozenblit and Keil (2002), reused a final time from Weeks 2, 3, 5 and 8. Full text from the authors’ lab. Their diagnostic-question method — asking someone to explain past the point where fluency runs out — is one concrete way to generate a genuine Challenge like the one Farhan faced. This is an empirical finding about explanatory depth, not a study of AI-assisted rehearsal; it supports the shape of the Challenge rung, not the specific hard case built around it.

What this leaves unresolved#

Farhan’s case is not decided, on purpose. Verdict A and Verdict B both remain live at the end of this lesson, and choosing between them would require evidence this exam was never designed to produce. What the course leaves you with instead is a way to say precisely what you can and cannot claim from what you have — which is a smaller thing than certainty, and a more honest one. There is no Week 13. This is where the course’s argument ends: not with a perfect test, but with a disciplined limit on what any test, however well built, is entitled to conclude.