A2 — Build a Bad Test

Design an instrument for something a peer claims to understand, administer it live, and report what it detected, what it missed, and what it fooled.

Due: 10 May 2027 · Weight: 30%

The brief#

Design a test for someone else’s claim to understand something, run it on them, and then tell us what your test actually measured.

This is the course’s secured, invigilated point: the administration happens live, in front of a witness, because a transcript afterward can’t settle whether the exchange was real. Pick a peer, a claim they’ve made about understanding something, and build a short instrument — drawing on the seven-rung ladder from Week 5 — meant to distinguish real command of the material from a fluent-sounding echo of it. Plant one deliberate trap: a fake explanation, a subtly wrong premise, a question the rungs shouldn’t be able to answer. Then run it.

The interesting failure here is not “the instrument was too easy.” It’s an instrument that returns a confident, wrong verdict — passing a fake, or failing something real — because that’s the same failure mode the fluent AI answer in Week 12 will exploit.

What you submit#

The instrument (questions, rungs targeted, what a pass and a fail look like), a record of the live administration, and an honest report of what the trap revealed. Name explicitly what a reader should not conclude from your result — what this particular instrument cannot rule out.