Week 11: Three People Is Not a Study

What a handful of observations can establish, what it cannot, and why "the evidence does not support a change" is a finding rather than a failure.

18 May 2027

Open the slides

Slides

What this lecture carries#

Existence proofs and rates. The single most useful distinction in small-n observation:

• “At least one person read the stall as a failure.” Established by one observation. Real, reportable, and often enough to act on.
• “Most people read the stall as a failure.” Not established by three observations, or by thirty, or by any number you will reach this semester.

Nearly every overreach in a student write-up — and in a good deal of published design writing — is the first sentence rewritten as the second. Watch for proportions: “two of three” is a count of who was in the room, not a rate.

Recording behaviour, not readings. “The user was confused” is a conclusion with its evidence deleted. “They moved the pointer to Cancel, stopped, and said ‘is it stuck or is it me’” is a record. Only the second can be re-examined next week when you disagree with yourself about what it meant.

Naming your sample precisely. Your participants are classmates. They know a stall is coming, because stalls were week 8. They have built this interface themselves. The useful move is not to apologise for this in general terms but to say what it specifically means: they will spot your indeterminate state faster than any real user, and they will be far more patient with it.

Deciding not to change something. Three legitimate responses to an observation: change it, leave it and say why, or leave it and name the test that would settle it. Manufacturing a change to demonstrate responsiveness is the failure mode this lecture exists to prevent. On Assignment 3 you are marked on whether your conclusions are proportionate to your evidence, never on how many people you watched or how many things you changed.

The alternative task. The structured walkthrough available in week 11’s studio is evidence about an interface’s internal consistency. It is not evidence about people. It is graded identically and it must never be written up as though somebody had been watched.

What it builds on#

Week 6 gave the class a weak result and asked them to feel why it was weak. Week 10 produced the claims register that tells you what is worth observing in the first place.

What it sets up#

A worked example#

One observation, written two ways. Only one of them is supportable.

A participant reaches for Cancel during the stall, stops, and says “is it frozen or is it just slow”.

Supportable: at least one person could not tell a stall from slow progress. That is an existence proof. One observation establishes it, it is reportable, and it is often enough to act on.

Not supportable: users cannot tell a stall from slow progress. No number of classmates gets you there. Nor does two of three, which is a count of who was in the room.

The first sentence licenses a change. The second licenses a claim, and you do not have one.

What this lecture cannot settle#

This lecture tells you what you cannot conclude. It cannot tell you what is worth watching for.

That comes from the claims register you built in week 10: the rows you marked kept with reason are the ones where your own judgement was doing the work, and they are where an outside reaction is most informative. Arriving on Thursday without having decided what to look at produces three pleasant sessions and no findings.

If you miss the hour#

The slides carry the argument in the order it was made, and this week’s reading is listed on the studio page. The studio on Thursday assumes the vocabulary above rather than re-teaching it, so read the deck before you arrive rather than after.