What if you didn’t know the test?#

The ladder assumption under examination#

Week 7 changed the form of a response; now we change advance notice. An Explanation or Challenge performance on Week 5’s ladder may have been rehearsed. An unforeseen follow-up tests that possibility, but the answer can still be constructed during the question. The new fictional example keeps this examination problem in view.

How to use this lesson#

Read at your own pace; no presenter is needed. Allow around 30 minutes. Have somewhere to write a few short answers, and be ready to write one before you see what comes next — that is part of how this week works.

You need no baking experience and no partner. All situations are fictional.

Where Week 7 left us#

Week 7 asked whether changing the form of an answer — a new audience, a new representation — was better evidence than repeating the original wording. It found a limit: reformatting can also be rehearsed, and a well-drilled explanation can be re-shaped into several forms without any of them testing something new.

That points at a variable none of the last three weeks isolated: whether the person knew, in advance, exactly what they would be asked.

Naming the variable: notice#

Call it notice — whether someone knows, before they perform, what question they will be asked and in what shape. Every activity so far in this course has given full notice: the task, the topic and the conditions were stated before the person began.

A prepared performance is produced with notice: the person could rehearse it, draft it, or fit it to a template they already had ready. That is not a flaw — most competent explanation in ordinary life is prepared. But it limits what the performance can tell an examiner.

The situation: a prepared explanation#

Elin is teaching themself to bake bread, and has written a short explanation of why dough rises, meaning to read it aloud to a flatmate later that evening. Elin has drafted it, reread it, and is confident in it. Nothing about the topic is a surprise: Elin chose it and had time to prepare.

The next slide is background you need to judge what Elin says — read it before Elin’s explanation, since Elin already has it.

Background: why dough rises#

Bread dough contains yeast, a living microorganism, and sugars from the flour. Yeast ferments those sugars and releases carbon dioxide gas as a by-product. Wheat flour mixed with water and worked by kneading develops gluten, a stretchy network of proteins that traps the gas rather than letting it escape. The trapped gas expands the dough, which is why it rises.

Technical background: King Arthur Baking Company, “The comprehensive guide to baking with yeast.” Link at the end.

Elin’s prepared explanation#

“The yeast eats the sugar in the flour and breathes out carbon dioxide — that’s what makes bubbles. Kneading builds up gluten, which is stretchy, so it stretches around the bubbles instead of popping. The dough puffs up because all those little bubbles are trapped inside it.”

Read it once as Elin’s flatmate would: is this accurate, complete, and does it explain why rather than just naming the steps?

A reasonable first judgement#

Checked against the background, Elin’s explanation is accurate: it names the mechanism (fermentation), the product (carbon dioxide), the structural change (gluten development) and the causal link between them (trapping rather than releasing the gas). It is fluent and uses the vocabulary correctly.

By the standards of earlier weeks, this already looks like decent evidence — Elin can produce an unprompted causal explanation, not just a recognised term. But every one of those sentences could have been rehearsed word for word.

What rehearsal leaves open#

A rehearsed explanation answers the question it was prepared for. It does not tell us whether Elin’s understanding extends past that exact sentence — to a version of the question Elin did not anticipate and could not have scripted in advance.

That is a different uncertainty from the ones earlier weeks raised. It is not about which resources were used (Week 6) or which form the answer took (Week 7). It is about whether the performance could have been produced without any causal model at all, purely by having the right paragraph ready.

The move: an unannounced follow-up#

If Elin’s explanation could be reproduced from memory alone, a follow-up question Elin has never considered should be harder to answer well — because there is no rehearsed sentence that already covers it. A good follow-up changes one condition of the original case and asks what happens.

This does not replace the prepared explanation; it is asked immediately after it, using the same topic, so any gap it reveals belongs to that explanation and not to an unrelated one.

Grounding this in a study#

Rozenblit and Keil ran an experiment with this exact structure. Sixteen graduate students rated their own understanding of everyday mechanisms (a zipper, a cylinder lock, and others), then wrote a detailed causal explanation, then answered an unforeseen diagnostic question — for the lock, explaining step by step how one could pick it.

What the comparison actually supports#

As Week 4 introduced, ratings after the diagnostic question were lower than before explaining. The change from just after explaining to after the diagnostic question was not statistically significant. This does not establish an additional effect of surprise alone.

Rozenblit & Keil (2002), Cognitive Science 26(5), 521–562, Study 1. An empirical finding from 16 participants at one university, not a general law.

Why the diagnostic question was not just a trick#

A natural objection is that the diagnostic question was simply harder, or unfair — testing knowledge the person never claimed to have. Rozenblit and Keil checked for this: after participants later read an expert description of the mechanism, they still judged their diagnostic-question-informed rating to have been accurate.

The authors interpret this as participants regarding the probes as relevant. It does not establish an additional effect of the diagnostic question, or isolate surprise as its cause.

Now, the follow-up#

Here is a generic follow-up template, applicable to most explanations: name one condition your explanation depends on, then ask what would happen if that condition did not hold. For Elin’s dough, an example would be: “What would happen if you left the salt out of the dough entirely?”

Apply the same move to your own explanation. Pick one thing your explanation assumed or relied on, and ask yourself what would change if it were different. Write your answer before reading on.

Worked example: Elin’s answer#

Asked what would happen without salt, Elin pauses, then answers: “I think the dough would still rise — maybe even more, since there’s nothing holding the yeast back. Salt’s mostly there for flavour, isn’t it? Though I have read that too much yeast activity too fast can make bread collapse rather than rise well, so maybe less salt isn’t automatically better.”

This was not in Elin’s prepared paragraph. Elin is visibly reasoning rather than reciting, correctly recalls that salt slows yeast fermentation, and connects that to a further consequence (over-fast fermentation can weaken the dough) that the original explanation never needed.

What this does and does not show#

Elin’s answer to the follow-up is better evidence than the prepared paragraph alone: it responds to a case Elin could not have scripted for, and it is right — salt does slow fermentation, which controls the rate of gas production and keeps the gluten structure from being overwhelmed before it can trap the gas properly.

But notice what we still cannot separate. Did Elin already know this before being asked, and simply had no occasion to say it in the original explanation? Or did Elin reason it out for the first time, live, while answering the question we just asked?

The problem surprise does not solve#

Both explanations fit what we observed. An unannounced follow-up is real progress — it rules out pure recitation, because there was nothing to recite. But the answer it produces is still a single, final performance, produced at one moment, and that moment already includes whatever thinking happened in response to being asked.

Surprise removes rehearsal. It does not remove the gap between the performance we can see and the state Elin was in before we asked the question.

Why this is the sharper version of an old problem#

Every fictional example this course has used shares one feature: we only ever see what someone says after we prompt them. Sam’s answer came after reading a source; Rowan’s came under three chosen conditions; Elin’s came after a follow-up designed to remove rehearsal.

In each case, the performance was shaped by the moment of producing it. An unannounced question shapes that moment less than a rehearsed one — but it still shapes it. What existed in Elin’s mind before any of our questions remains something we infer, not something we observed.

Consolidate: what surprise buys you, and what it costs#

What it buys: a follow-up the person could not have prepared for is much harder to answer by reciting a script, so a good answer to it is stronger evidence of a working causal model than a good answer to the original, anticipated question alone.

What it costs: the answer is now shaped by reasoning that may have happened live, during the question itself, which is very hard to distinguish from reasoning that existed beforehand and was simply never asked for.

A short record#

Write three sentences about the follow-up you answered for your own topic:

1. Observation: “Asked what would change if [condition] did not hold, I said…”
2. Interpretation: “This suggests I can…”
3. Limit: “It does not show whether I already had this reasoning ready, or built it while answering.”

A useful limit names that specific ambiguity, not a general disclaimer that you might be wrong.

Compare with this example record#

Observation: Asked what would happen without salt, Elin correctly named its slowing effect on fermentation and connected it to gluten structure, without having prepared that answer.

Interpretation: Elin’s causal model extends past the rehearsed explanation, at least to this one unforeseen case.

Limit: We cannot tell whether Elin already held this connection before being asked, or assembled it during the answer. Either way, it is evidence about Elin’s model now, not about the state Elin was in beforehand.

What Week 8 hands forward#

If we cannot always separate reasoning-on-the-spot from prior knowledge after the fact, one option is to stop working only from after-the-fact performances. What if we captured what someone thought before any intervention — a prediction, made and recorded, that we can compare against what they do next?

That is next week’s problem: the prior state, made explicit before checking anything.

What to take from Week 8#

A question the person could not anticipate is harder to answer by rehearsal, and a good answer to one is stronger evidence than a good answer to a question they saw coming. Use unannounced follow-ups deliberately, changing one condition the original explanation depended on.

But the resulting performance is still a single snapshot, produced in response to being asked. It narrows what a script could explain; it does not tell you what existed before the question.

Read further: surprise and diagnostic questioning#

Rozenblit & Keil (2002). “The misunderstood limits of folk science: an illusion of explanatory depth.” Cognitive Science, 26(5), 521–562, Study 1. Ask: why might people judge their post-diagnostic-question rating as accurate, even after learning more? Read the paper.

This reading deepens the lesson; you can complete the activities without it.

Sources and your next step#

Technical background: King Arthur Baking Company: The comprehensive guide to baking with yeast

Elin and the dialogue above are fictional teaching examples, not reported research results.

For the complete written activity and a record to keep, open the Week 8 investigation. The lecture notes offer a continuous reading version.