Evidence changes with conditions#

Retrieve your Week 3 record · 2 minutes#

Bring back your annotated reconstruction and retained judgement from Week 3. Keep the original unchanged. After reading Rowan’s three responses, write two sentences: which gap in your reconstruction would each condition reveal or conceal, and does that change your original judgement? If you have no record, use Priya’s worked reconstruction from Week 3, or write a short source-closed reconstruction from this week’s lock explanation before proceeding.

Worked comparison#

For example, an omitted causal step may become visible when explaining to a novice, while an open reference may supply that step. A stronger supported answer therefore does not establish that the earlier gap disappeared. If your answers agree across conditions, record that agreement and what remains untested.

How to use this lesson#

Read at your own pace; no presenter is needed. Allow about 40 minutes, and have somewhere to write.

At a Pause and write slide, stop and attempt the question before you advance. The slides after it give a worked response with its reasoning and its limits.

You need no prior knowledge of locks, no lock, no tools and no subscription to anything. Every person, situation and response in this lesson is a fictional teaching example.

Where the course has got to#

Week 1 separated what you observed from what you inferred, and put the conditions an answer was produced under into the record.

Week 2 found that a fluent explanation can conceal gaps. Week 3 found that taking the source away reveals more, but that a reconstruction can still be rehearsed or assisted.

Each of those weeks examined one performance at a time.

This week we compare#

Today you get three performances by the same person on the same concept.

More evidence turns out not to help in the way you would expect. The three pieces do not agree, and nothing in them tells you which to believe.

One fictional person, Rowan, will account for one mechanism three times, under three different conditions.

Why the same lock as Week 3#

The mechanism is a pin tumbler lock: the kind in most front doors. Week 3 used the same lock, deliberately.

This week’s claim is that one person on one concept can draw different judgements. If the object changed, there would be an obvious rival explanation for any difference we found.

If you have not done Week 3, you lose nothing. The mechanism is explained from scratch over the next few slides.

The lock, part one: the outside parts#

A pin tumbler lock has a fixed outer casing. Inside it sits a cylinder that can rotate, called the plug.

The key goes into a slot in the front of the plug, called the keyway. At the back of the plug is a cam: a small arm that pulls the bolt out of the door frame when the plug turns.

So the key has one job. It has to make the plug free to rotate.

The lock, part two: the pin stacks#

Sitting across the boundary between plug and casing is a row of small stacks, usually five or six of them.

Each stack has three things in it. A key pin at the bottom, resting in the plug. A driver pin above it. And a spring above that, pressing the whole stack downward.

Technical background throughout: Wikipedia, “Pin tumbler lock.” Link at the end. It grounds the example only.

The lock, part three: the shear line#

The shear line is the boundary itself: the circular join where the outside of the plug meets the casing around it.

It is not a component you could pick up and hold. It is a line, and it is the line the plug has to be able to turn along.

Everything else in this lesson depends on what is sitting across it.

Stuck, and free to turn#

Nothing crosses the line, so the plug turns, the cam moves, and the bolt withdraws.

What a wrong key does#

A wrong key still lifts the stacks. It lifts them by the wrong amounts.

Lift one stack too little and its driver pin is still hanging down inside the plug. Lift one too far and the key pin itself is pushed up into the casing.

Either way something is crossing the shear line, and the plug will not turn.

There is no partly open#

Notice what that means. The lock does not grade you, and it does not open a little.

Anything crossing that line, anywhere along the row of five or six stacks, is enough to stop the plug dead.

Hold on to this. It becomes the whole of condition (c).

What we mean by a condition#

A condition is a feature of the situation an answer was produced in, which could itself change the answer.

What was available to consult. Who the answer was for. How much warning there was. What kind of question was asked at all.

Week 1 asked you to record conditions. This week explains why they are not bookkeeping.

Conditions are not difficulty settings#

The tempting picture is that a hard condition gives you the same evidence with more noise in it. That picture is wrong.

Removing a reference does not strip assistance away to reveal the person underneath. It adds a demand on memory, which is a different thing to have evidence about.

The condition helps produce the performance. Change it and you have collected evidence about something else.

Three conditions, one lock#

Held constant: the person, the mechanism, and the same afternoon.

(a) Reference visible. Explain how the lock works, with the text in front of you.

(b) From memory, to a novice. Explain it to someone who has never thought about how a key works, with nothing to consult.

(c) A prediction about a changed case. One cut on a working key has been filed too shallow. What happens, and why?

Condition (a): the question put to Rowan#

Rowan has the reference text open on the table. It contains exactly what you have just read about the lock, and nothing else.

The question: “Explain how a pin tumbler lock works.”

Nothing is timed and nothing is taken away. Worth noticing in advance: we have no independent record of how much of the reference Rowan actually looked at.

Rowan under condition (a)#

“A pin tumbler lock has a plug that rotates inside a fixed casing. The key goes into the keyway at the front of the plug.

Along the boundary between the two there are five or six stacks of pins, and each stack is a key pin at the bottom, sitting in the plug, a driver pin above it, and a spring above that pressing down.”

Rowan under condition (a), continued#

“That boundary is called the shear line. With no key in, the springs push the stacks down so the driver pins cross the shear line into the plug, and that is what stops it turning.

The key pins are different lengths, and the key is cut to match them. The correct key lifts each key pin so that the junction between the key pin and the driver pin ends up exactly at the shear line.”

Rowan under condition (a), ending#

“Then nothing is crossing the boundary, the plug turns, and the cam at the back withdraws the bolt.

A wrong key lifts the pins by the wrong amounts, so something still crosses the shear line and it stays locked.”

That is accurate, complete, correctly ordered, and every term is used properly.

Condition (b): the question put to Rowan#

The reference is closed and removed. The listener is a friend who has never once wondered what happens inside a key.

The question: “Explain to them how a key opens a lock.”

Rowan now has two jobs at once: recover the mechanism without the text, and recast it for someone with no vocabulary for it. Hold that double demand in mind.

Rowan under condition (b)#

“OK, so — you know the zigzag notches along the top edge of a key? They are not decoration. They are a code, and the lock is the thing that reads it.

Inside the lock there is a barrel that has to spin round for the door to open.”

Rowan under condition (b), continued#

“Normally it cannot, because there is a row of little pins, five or six of them, sticking down through the join between the barrel and the metal around it. A pin sitting across a join is a bolt. Nothing is turning.

Every pin is a different length and every notch is a different depth. The right key lifts each pin by exactly its own amount, so all of them come flush with that join at the same moment.”

Rowan under condition (b), ending#

“Nothing is sticking across it any more, the barrel spins, and a little arm at the back pulls the bolt out of the frame.

A wrong key does lift the pins. It just lifts them to the wrong heights, so one is always left poking across the join, and it stays shut.”

Condition (c): the question put to Rowan#

Still nothing to consult. The question is one Rowan has not been asked before:

“Take a key that works this lock. One single cut on it, say the third one along, has been filed a little too shallow. Every other cut is correct. What happens when you put it in and turn it, and why?”

This asks for a prediction about a mechanism that has been changed.

Rowan under condition (c)#

“Sorry — do you want me to guess, or is there a right answer? … Right. OK.

Too shallow means it does not lift that pin far enough. So that one is sitting low. The bit of it that is supposed to end up level with the join is below the join instead, so something is still crossing, at that one position.”

Rowan under condition (c), continued#

“And — this is the bit I am least sure about — I think it does not matter that the other five are fine. One is enough.

Because the barrel is one piece. It is not five separate locks that each open a fifth of the way. So it does not half turn or feel loose. It just does not turn, the same as a completely wrong key.”

Rowan under condition (c), ending#

“Hold on. Would you even be able to tell this key from a totally wrong one, from the outside?

I do not think you would. That feels wrong to me but I cannot see what would make it false.”

Rowan cannot have read this answer anywhere, because this key does not exist. Measured against the mechanism above, it is also correct.

Worked response: the case that Rowan understands it#

Response (a) is correct and complete, so nothing Rowan says about the mechanism is confused.

Response (b) then recovers the central relation without the text and in Rowan’s own words: everything must clear one boundary at the same moment, and a pin across a join is a bolt.

That phrasing appears nowhere in the reference, so it was not lifted from it.

The case that Rowan understands it, continued#

Response (c) predicts a case that could not have been memorised, and reaches the consequence that actually matters.

One wrong stack is sufficient, because the plug is a single object rather than five independent ones. That is a causal relation, not a part name.

On this reading, Rowan holds the mechanism and can carry it between tasks and audiences.

Worked response: the case that Rowan does not#

Response (a) tracks the reference’s order and vocabulary closely enough that a sceptic can read it as well-organised copying.

Response (b) then quietly loses the two-pin stack. Rowan describes one pin per position being lifted “flush with the join.” What has to reach the shear line is the junction between two pins, not a pin.

The case that Rowan does not, continued#

Watch what follows. With one pin per position, lifting it too far would carry it clear of the join and the lock would open. Rowan’s model cannot fail by over-lifting — and sure enough, Rowan’s wrong key only ever lifts too little.

Response (c) then repeats the same vagueness about which part reaches the boundary, hesitantly. On this reading Rowan has a picture, not a mechanism.

A sharper version of that objection#

Rowan read the reference in condition (a), and it says that anything crossing the shear line anywhere along the row is enough to stop the plug.

So (c)‘s headline conclusion was already available in print. What the reference does not supply is Rowan’s reason for it: that the plug is one object, so there is no partial turn and no way to feel the difference from outside.

If you want to defend (c), that is the part to defend.

Both cases were built from the same three texts#

Neither reading is a misreading. Each is supported by things Rowan actually said, and neither ignores anything.

So if you now go back and look harder at Rowan, you will not settle it. There is no further detail in these three responses that decides between the two cases.

Evidence is produced by a situation#

This is the week’s central idea, so here it is plainly.

A performance is not a window onto a person, clearer or cloudier depending on how hard you look through it. The condition helps produce what you see.

So “does Rowan understand the lock?” is underspecified as it stands. Well enough to do what, under which conditions?

Why condition (c) carries the most#

A prediction about a modified mechanism is the only one of the three that cannot be satisfied by holding the parts in the right order.

To answer it you need to know what each part does to the next: cut depth sets pin height, pin height decides whether something crosses the shear line, and the plug is one object so one crossing is decisive.

Part names will not produce that. Causal relations will.

Why condition (c) is also the most fragile#

Look again at how Rowan opened it: “do you want me to guess, or is there a right answer?”

Somebody can hold the whole mechanism and still stall there, because they have never been asked to predict anything and do not know what kind of answer is wanted, how much detail counts, or whether hedging is allowed.

The genre of the task is itself unfamiliar.

The double edge#

So a weak answer under (c) is ambiguous between two completely different explanations: no causal model, or no experience of being asked. The transcript will not separate them.

The condition that produces the most informative evidence when it works is also the one most easily disturbed by something that has nothing to do with locks.

Worked response: two defensible choices#

Trust (a). It is the only condition where Rowan can be checked against the mechanism without memory interfering. Objection: it cannot distinguish understanding from careful reading, which is the distinction we wanted.

Trust (b). The source is gone and the audience is new, so whatever survives is Rowan’s own. Objection: an explanation rehearsed for a novice is exactly the kind that can be fluent and wrong — and this one was.

Worked response: the third, and the catch#

Trust (c). It is the only condition whose answer could not have existed in advance, so a good answer cannot be explained away. Objection: it is also the most likely to fail for reasons unrelated to the lock.

Each is a real answer with a real cost. Notice what none of them is: a reason that would settle the disagreement for someone who started from a different one.

One thing worth noticing across all three#

Rowan is vague about the same thing twice. In (b) it is one pin coming “flush with the join.” In (c) it is “the bit of it that is supposed to end up level with the join.”

Both times the two-pin stack has collapsed into one.

Why a repeated gap is worth more#

A gap that appears under one condition might belong to the condition — a slip of phrasing for a novice, a hesitation under an unfamiliar question.

A gap that survives a change of condition is much harder to blame on the situation.

That leverage comes from comparing performances, not from examining one of them harder. It is also small: two observations, from two conditions that both lacked the reference.

What if your judgements were consistent?#

Some readers give the same verdict under all three conditions and feel no tension. That is a real result, and it should be taken seriously rather than treated as a failure to notice something.

But be precise about what it establishes. It shows you applied something stable. It does not show the something was a good criterion, or that you could state it.

A check on consistency#

The common version is that you formed a view under one condition and read the other two through it, without ever registering that you had chosen a condition.

So write down what evidence would have changed your verdict.

If nothing would have, your consistency came from a commitment rather than from the evidence.

The limitation this week produces#

Held against three conditions, intuitive judgement is not stable. It moves with the situation, and it moves without announcing that it has.

That is not a reason to distrust intuition entirely. It produced real observations here: the missing second pin, the hedged opening, the independently phrased relation.

The problem is not the observing. It is the weighing.

An empirical echo#

In a 2002 study of how well people think they understand everyday things, Rozenblit and Keil asked participants to rate their own understanding on a scale of 1 to 7. One of their eight test items, as it happens, was a cylinder lock.

Ratings fell after participants wrote a step-by-step causal explanation, and again after a question probing the mechanism.

The same person, the same hour; the rating moved because the task changed.

The echo, with its limits#

Study 1 was 16 graduate students at one university, and these are self-ratings — self-report, not independent evidence of what anyone could produce.

And the drop after writing an explanation was not significantly different from the drop after the probe. Conditions matter, but which ones matter is an empirical question, not something intuition settles in advance.

Rozenblit & Keil (2002), Cognitive Science 26(5), 521–562, Study 1. An empirical finding, not a theory. Extending it to an outside examiner is our adaptation, not their claim.

What is missing is a stated rule#

Notice what we could not do at any point today. We could not say, in advance and out loud, what would make one of Rowan’s responses stronger evidence than another.

Without that, a disagreement collapses into two people asserting which condition mattered, with no way for either to be shown wrong.

Keep a short record#

Before you finish, write four lines:

1. The evidence: which of Rowan’s three responses moved your verdict most.
2. The condition: what it made possible, and what it ruled out.
3. The rule: finish “I counted that as stronger evidence because…”
4. The limit: one thing your verdict does not establish about Rowan.

Line 3 matters most, and is most likely to be hard to write.

Where Week 5 goes#

If line 3 was difficult, that difficulty is the finding. An unstated rule cannot be checked by anyone, including you.

Week 5 builds the course’s first instrument: a stated ladder of performances, from recognising something up to connecting it to other things, with each rung defined well enough that two people can apply it and locate their disagreement.

This week does not build it. Naming what is missing is the result.

Read further: the empirical source#

One warning before you read it: their probe question for the lock asked how one could defeat it. Our condition (c) deliberately does not, and this course gives no method for defeating a lock.

Read further: the argument#

Week 1 used this to argue that a reference does not automatically disqualify a performance. Ask the harder version: what would have to be true of Rowan’s use of the reference in condition (a) for it to count as part of the thinking — and could you tell from the transcript alone?

Sources and your next step#

Technical grounding for the lock: Wikipedia, “Pin tumbler lock”. It supports the mechanism only, and is not evidence for any claim about understanding or learning.

Rowan, the three conditions and all three responses are fictional teaching examples written for this lesson. They are not transcripts and not research data.

For the written investigation and a record to keep, open the Week 4 investigation. The lecture notes give a continuous reading version.