Week 8: What if you didn't know the test?
Watch a fictional baker's prepared explanation meet an unannounced follow-up question, and find out what surprise can reveal about a causal model that rehearsal cannot.
12 April 2027
The ladder assumption under examination#
Week 7 changed the form of a response; now we change advance notice. An Explanation or Challenge performance on Week 5’s ladder may have been rehearsed. An unforeseen follow-up tests that possibility, but the answer can still be constructed during the question. The new fictional example keeps this examination problem in view.
What does surprise reveal that preparation can hide?
This is a self-guided lesson. Both these notes and the slide deck contain the explanations you need; no presenter is assumed. Allow about 30 minutes. Have somewhere to write short answers, and be ready to write one before reading further — that is part of how the lesson works. No baking experience and no partner are required. The people below are fictional.
What you will learn#
By the end, you should be able to define notice as a property of a task, explain why a prepared performance under-determines whether someone has a working causal model, describe how an unannounced follow-up narrows that uncertainty, and explain the specific limit that even a strong answer to a surprise question still leaves open.
1. Where Week 7 left us#
Week 7 asked whether changing the form of an answer — a new audience, a new representation of the same idea — gave better evidence than repeating the original wording. It found a limit: reformatting can itself be rehearsed, and a well-drilled explanation can be reshaped into several forms without any of them testing something genuinely new. Week 6, before that, asked whether the resources available changed what a performance could show. Neither week isolated a variable that both of them quietly assumed: whether the person knew, in advance, exactly what they would be asked.
2. Naming the variable: notice#
Call it notice — whether someone knows, before they perform, what question they will be asked and in what shape. Every activity so far in this course has given full notice: the task, topic and conditions were stated before the person began. Sam knew the question about the outage; Rowan knew all three of the conditions being asked for.
A prepared performance is one produced with notice: the person could rehearse it, draft it, or fit it to a template they already had ready. That is not automatically a flaw — most competent explanation in ordinary life is prepared. But it limits what the performance alone can tell an examiner about whether a causal model exists behind it, or whether a well-chosen sentence was simply ready in advance.
3. The situation: a prepared explanation#
Elin is teaching themself to bake bread and has written a short explanation of why dough rises, meaning to read it aloud to a flatmate that evening. Elin has drafted it, reread it, and is confident in it. Nothing about the topic is a surprise: Elin chose it and had time to prepare.
Bread dough contains yeast, a living microorganism, and sugars from the flour. Yeast ferments those sugars and releases carbon dioxide gas as a by-product. Wheat flour mixed with water and worked by kneading develops gluten, a stretchy network of proteins that traps the gas rather than letting it escape. The trapped gas expands the dough, which is why it rises.
Technical background: King Arthur Baking Company, ["The comprehensive guide to baking with yeast"](https://www.kingarthurbaking.com/blog/2023/05/10/guide-to-baking-with-yeast).
Elin’s prepared explanation, verbatim: “The yeast eats the sugar in the flour and breathes out carbon dioxide — that’s what makes bubbles. Kneading builds up gluten, which is stretchy, so it stretches around the bubbles instead of popping. The dough puffs up because all those little bubbles are trapped inside it.”
4. A reasonable first judgement, and what it leaves open#
Checked against the background above, Elin’s explanation is accurate: it names the mechanism (fermentation), the product (carbon dioxide), the structural change (gluten development) and the causal link between them (trapping rather than releasing the gas). By the standards of earlier weeks this already looks like decent evidence — an unprompted causal explanation, not a recognised term repeated back. But every sentence of it could have been rehearsed word for word, since Elin wrote and reread it before saying it.
A rehearsed explanation answers the question it was prepared for. It does not tell us whether Elin’s understanding extends past that exact sentence, to a version of the question Elin never anticipated and could not have scripted in advance. This is a different uncertainty from the ones the last two weeks raised: it is not about which resources were used, or which form the answer took. It is about whether the performance could have been produced from a ready paragraph alone, with no causal model doing any work behind it.
5. The move: an unannounced follow-up#
If Elin’s explanation could be reproduced from memory alone, a follow-up question Elin has never considered should be harder to answer well, because there is no rehearsed sentence that already covers it. A good follow-up changes one condition of the original case and asks what happens — asked immediately after the prepared explanation, on the same topic, so any gap it reveals belongs to that explanation rather than to something unrelated.
Rozenblit and Keil’s Study 1 used exactly this structure. Sixteen graduate students at Yale rated their own understanding of everyday mechanisms — among them a zipper and a cylinder lock — then wrote a detailed, step-by-step causal explanation, then answered an unforeseen diagnostic question requiring critical knowledge of the mechanism (for the lock, explaining step by step how one could pick it). Self-ratings after the diagnostic question were lower than before the explanation (p < .002). The additional change from post-explanation to post-diagnostic was not statistically significant. As Week 4 introduced, this study supports caution about self-assessment; it does not establish an extra effect of the diagnostic question alone.
A natural objection is that the diagnostic question was simply harder or unfair, testing knowledge the person never claimed. Rozenblit and Keil checked for this directly: after participants later read a brief expert description of the mechanism, they still judged their diagnostic-question-informed rating to have been accurate. That is the authors’ own reading of that later rating, not an inference we are adding. This supports the authors’ interpretation that participants regarded the probes as relevant. It does not show a statistically reliable additional drop after the diagnostic question, or isolate surprise as its cause.
Rozenblit & Keil (2002), Cognitive Science 26(5), 521–562, Study 1: 16 graduate students at one university. An empirical finding, not a general law, and self-report about self-generated ratings, not independent evidence of what any one participant could produce.
6. Pause and write: set your own surprise#
Allow three minutes. Choose something you can already explain fluently — a process, a mechanism, how something works. Do not pick something you would need to look up. Write a short explanation, two to four sentences, as if a friend had asked you to describe it.
Do not read the next paragraph until you have written it; the point depends on genuinely not having seen the follow-up first.
The follow-up, once you have written your explanation
Here is a generic follow-up template, applicable to most explanations: name one condition your explanation depends on, then ask what would happen if that condition did not hold. For Elin’s dough, an example is: “What would happen if you left the salt out of the dough entirely?” Apply the same move to your own explanation — pick one thing it assumed or relied on, and ask yourself what would change if it were different. Write your answer before reading on.
7. Worked example: Elin’s answer, and what it shows#
Asked what would happen without salt, Elin pauses, then answers: “I think the dough would still rise — maybe even more, since there’s nothing holding the yeast back. Salt’s mostly there for flavour, isn’t it? Though I have read that too much yeast activity too fast can make bread collapse rather than rise well, so maybe less salt isn’t automatically better.” This was not in Elin’s prepared paragraph. Elin is visibly reasoning rather than reciting, correctly recalls that salt slows yeast fermentation, and connects that to a further consequence — over-fast fermentation can weaken the dough — that the original explanation never needed.
Elin’s answer to the follow-up is better evidence than the prepared paragraph alone: it responds to a case Elin could not have scripted for, and it is correct — salt does slow fermentation, which controls the rate of gas production and keeps the gluten structure from being overwhelmed before it can trap the gas properly.
But notice what we still cannot separate. Did Elin already know this before being asked, and simply had no occasion to say it in the original explanation? Or did Elin reason it out for the first time, live, while answering the question we just asked? Both explanations fit exactly what we observed.
8. The problem surprise does not solve#
An unannounced follow-up is real progress: it rules out pure recitation, because there was nothing prepared to recite. But the answer it produces is still a single, final performance, produced at one moment, and that moment already includes whatever thinking happened in direct response to being asked. Surprise removes rehearsal. It does not remove the gap between the performance we can see and the state Elin was in before we asked the question.
This is the sharper version of a problem that has run through every fictional example in this course. We only ever see what someone says after we prompt them. Sam’s answer came after reading a source; Rowan’s came under three chosen conditions; Elin’s came after a follow-up designed specifically to remove rehearsal. In each case, the performance was shaped by the moment of producing it. An unannounced question shapes that moment less than a rehearsed one — but it still shapes it. What existed in Elin’s mind before any of our questions remains something we infer, not something we directly observed.
9. What surprise buys you, and what it costs#
What it buys: a follow-up the person could not have prepared for is much harder to answer by reciting a script, so a good answer to it is stronger evidence of a working causal model than a good answer to the original, anticipated question alone.
What it costs: the answer is now shaped by reasoning that may have happened live, during the question itself — reasoning that is very hard to distinguish from reasoning that existed beforehand and was simply never asked for until now.
10. Keep a short record#
Write three sentences about the follow-up you answered for your own topic:
| Part of the record | Example |
|---|---|
| Observation | Asked what would happen without salt, Elin correctly named its slowing effect on fermentation and connected it to gluten structure, without having prepared that answer. |
| Interpretation | Elin’s causal model extends past the rehearsed explanation, at least to this one unforeseen case. |
| Limit | We cannot tell whether Elin already held this connection before being asked, or assembled it during the answer; either way, this is evidence about Elin’s model now, not about the state Elin was in beforehand. |
A useful limit names that specific ambiguity, not a general disclaimer that the observer might be wrong.
11. Apply the lesson yourself#
Use the Week 8 investigation to set your own prepared explanation, meet an unannounced follow-up, and keep a record of what it did and did not settle. Work alone; a partner variant is offered there too, with its own limits stated.
Before you finish, explain these distinctions in your own words:
- A prepared performance versus one produced without notice.
- What an unannounced follow-up rules out, and what it still cannot separate.
- Why a strong answer to a surprise question is still only evidence about the moment it was produced.
Reading with a purpose#
Empirical finding: Rozenblit, L., & Keil, F. (2002). “The misunderstood limits of folk science: an illusion of explanatory depth.” Cognitive Science, 26(5), 521–562. Study 1: 16 graduate students at Yale University. Read the paper. Ask: why might participants judge their post-diagnostic-question rating as accurate, even after later reading an expert description that gave them more information? This reading deepens the lesson; you can complete the activities without it.
What this leaves unresolved#
An unannounced follow-up narrows what rehearsal alone could explain, but the resulting performance is still a single snapshot, shaped by the act of being asked. It cannot tell us what existed before the question. Next week asks what we could learn instead from a prior state — a prediction or explanation committed to and recorded before any intervention — and how that changes what a later performance can be compared against.