Week 11: Algorithms as modern oracles

When a recommendation algorithm "gets you," is it reading you personally, or reading everyone who looks statistically like you?

A modern recommendation algorithm and a horoscope column are solving a suspiciously similar problem: produce output that feels individually tailored, using a process that is actually running the same computation for a huge population at once. The horoscope does this with twelve broad buckets and deliberately vague language. A recommendation system does it with millions of behavioural data points and a genuinely predictive statistical model — and yet the subjective experience, “it’s like it knows me,” lands with comparable force in both cases.

The mechanism, unlike a horoscope’s, really is discovering something: patterns in your past clicks, purchases and watch-time do predict your future behaviour better than chance, often uncomfortably well. That’s a real epistemic difference from astrology, and worth taking seriously as such. But the feeling of being personally known is doing something separate from the accuracy — it’s the same feeling produced by a Barnum statement in week 6, just backed this time by a genuinely working prediction engine instead of a vague paragraph.

This raises the question the final project asks you to sit with directly: once a system is accurate enough, does the distinction between “oracle” and “tool” still matter? An algorithm that predicts your mood, your next purchase, or your risk of leaving a relationship isn’t claiming mystical insight — but if it’s right often enough, are you treating it any differently than the ancient reader who was occasionally right by chance?

How the trick actually works: collaborative filtering#

The dominant technique behind most recommendation algorithms is collaborative filtering, and it’s worth stating plainly because the name sounds more mysterious than the idea is: find other people whose past behaviour looked like yours, and recommend what they did next. The system never has to understand a song, a film or a product at all — it only needs enough shared behavioural history across enough users to notice that people who liked A and B usually also liked C. Nobody examined your specific taste; a correlation across millions of other profiles stood in for that examination, and it works often enough that the distinction barely shows up in the output.

“My phone is listening to me”#

This is one of the most widely reported modern folk beliefs about algorithmic prediction, and it’s worth treating carefully rather than dismissing outright. Independent investigations by security researchers and journalists have repeatedly gone looking for evidence of apps secretly using a phone’s microphone to feed advertising systems, and have repeatedly come back empty — the claim doesn’t hold up under direct technical scrutiny. What does hold up is that advertising networks share cross-app tracking data, browsing history and location on a scale most users don’t fully picture, and combined with the base-rate reasoning from week 2 — you think about far more products than you notice ads for, and only the hits get remembered as “it was listening” — the feeling has a fully mundane explanation that doesn’t require any microphone at all. The belief persists not because the mechanism has been found, but because the alternative explanation (silent, extensive tracking) is somehow less unsettling to imagine than the literal one.

The newest voice in the chain#

The newest development compresses several weeks of this course into one product: apps and services now use large language models to generate horoscope-style or tarot-style interpretive text on demand, rather than drawing from a small fixed pool of pre-written lines like week 6’s horoscope column or this week’s own demo oracle. The underlying mechanism is still the Barnum effect from week 6 — plausible-sounding, personalised-feeling text built to fit almost any reader — but the combinatorial variety is now enormous, generated fresh per request instead of recycled from a shelf of stock phrases. It’s the oldest trick in the course wearing the newest technology, and arguably the hardest version to spot precisely because no two readings look alike.

Four oracles, two questions#

Placed side by side, the systems this course has covered separate cleanly along two axes: does the system draw from a fixed pool or a learned model, and is its mechanism disclosed or opaque to the person using it?

• I Ching (week 8): fixed pool of 64 texts, mechanism fully disclosed (a visible coin toss).
• Horoscope (week 6): fixed pool of Barnum statements, mechanism typically undisclosed.
• Recommendation algorithm: learned model built from millions of users, mechanism almost always opaque to the user.
• AI-generated interpretation: a learned model generating from an effectively unbounded space, mechanism opaque unless a provider chooses to explain it.

Disclosure, not accuracy, is the axis this course’s own interactive components have been built around from week 1 onward — and it’s the one axis a system’s designer, rather than its user, actually controls.

Further reading#

• Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016).

Ask the algorithm#

Get a recommendation, then read the mechanism note — it’s the most literal, least mystical explanation in the entire course, and arguably the most unsettling one.