Week 10: Cognitive biases

Confirmation bias and illusory correlation, named properly — the two mechanisms that have been running underneath every week so far.

This week gives the underlying mechanism a proper name, three weeks after first meeting it in disguise. Confirmation bias is the tendency to notice and remember evidence that fits a belief you already hold, and to overlook or explain away evidence that doesn’t — it’s the reason the hit from a horoscope sticks and the miss doesn’t. Illusory correlation is the related tendency to perceive a statistical relationship between two things that aren’t actually related, particularly when one of them is rare or emotionally charged — it’s why a “lucky” pair of socks survives a decade of losses and gets credit for the one big win.

Neither bias is a character flaw or a sign of poor reasoning. Both have been demonstrated repeatedly in controlled experiments, in subjects with no particular investment in astrology, tarot or omens — they are default settings of ordinary cognition, not symptoms of gullibility. That is the important reframe for this stage of the course: everything covered in weeks 3, 5, 6, 7, 8 and 9 can be fully explained without assuming anyone involved is being irrational. They are being exactly as rational as human cognition ordinarily is — which turns out to be less rational, in specific and well-mapped ways, than intuition suggests.

This is also the week to look back at your own Pattern Diary with fresh eyes. Several entries probably describe something that felt meaningfully connected. Read them again asking: what did I notice, what did I link it to, and would I have remembered the miss as clearly as I remembered the hit?

Three more names, three more studies#

Confirmation bias and illusory correlation don’t work alone. Three related mechanisms round out the toolkit this week hands you.

Motivated reasoning is reasoning bent, often unconsciously, toward a conclusion you already want. Albert Hastorf and Hadley Cantril’s classic 1954 study “They Saw a Game” showed Princeton and Dartmouth students genuinely different footage of the same rough football match between their two teams — Princeton students saw Dartmouth committing roughly twice as many infractions as Dartmouth students saw their own team commit. Neither group was lying; each was watching the same recording through a filter shaped by which outcome it wanted to be true.

Hindsight bias is the “I knew it all along” effect: once an outcome is known, people reliably overestimate how predictable it had been beforehand. Baruch Fischhoff’s 1975 experiments gave subjects a description of a historical event and asked some to predict its outcome in advance, others to recall their own (fabricated) prior prediction after learning what actually happened — the second group consistently “remembered” having predicted the real outcome, even when their genuine prior guess was on record and said otherwise. Every “this was obviously coming” reaction to an event you didn’t in fact predict is this bias doing its ordinary work.

Self-fulfilling prophecy is a belief that changes the very outcome it predicted. Robert Rosenthal and Lenore Jacobson’s 1968 “Pygmalion in the classroom” study told teachers that certain students (chosen at random) were about to show a spurt of academic promise — and those students’ performance measurably improved over the year, driven by nothing but the teachers’ altered expectations and behaviour. A prediction can come true by changing how the predictor acts, with no need for it to have been accurate to begin with.

The case that hinges on what isn’t there: survivorship bias#

Selection effects — where the data you can see is systematically unlike the data you can’t — get their sharpest illustration from Abraham Wald’s work for the U.S. military in the Second World War. Engineers wanted to know where to add armour to bombers, and had detailed maps of bullet-hole locations on planes that returned from missions: wings and fuselage riddled, engines comparatively untouched. The obvious conclusion — armour the bullet-heavy areas — was exactly backwards. Wald pointed out that the returning planes were a filtered sample: planes hit in the engine mostly didn’t come back at all. The right place for armour was precisely the unmarked areas on the surviving planes, because a hit there was the kind that prevented a plane from ever entering the dataset. Jordan Ellenberg’s How Not to Be Wrong (2014) retells this case as the canonical example of a selection effect — a warning that “look at the pattern in the data” is only as good as knowing what never made it into the data in the first place.

Naming the trick doesn’t settle the score#

Here is the point this week most needs to make explicitly, because it’s easy to walk away from six named biases thinking weeks 3, 5, 6, 7, 8 and 9 have now been debunked. They haven’t — not by this move alone. Explaining the psychological mechanism behind a belief (why someone finds astrology compelling, why a dream feels prophetic) is a different claim from showing the belief is false. Treating “I can explain why you believe this” as equivalent to “therefore you’re wrong” is its own reasoning error, sometimes called the genetic fallacy — the truth of a claim doesn’t depend on the psychological route someone took to reach it. What this week’s mechanisms actually establish is narrower and more useful: they explain why a belief can feel overwhelmingly well-supported regardless of whether it’s true, which means “it feels obviously true” stops being usable as evidence either way. The belief still has to be checked on its own terms.

Further reading#

• Albert H. Hastorf & Hadley Cantril, “They Saw a Game: A Case Study” (1954).
• Baruch Fischhoff, “Hindsight ≠ Foresight” (1975).
• Robert Rosenthal & Lenore Jacobson, Pygmalion in the Classroom (1968).
• Jordan Ellenberg, How Not to Be Wrong: The Power of Mathematical Thinking (2014).

Look, then decide#

Look at the two datasets honestly before reading the reveal — the more confidently you pick one, the more useful the reveal will be.