Week 9: A metric that cannot fail is noise

Nine different builds, 34 kills each. A metric that gives every subject the same score is not measuring them, and it will not tell you so.

4 May 2027

Is it me or is it the game? The metric may not be able to say. Extra damage took the runs that kill the first boss from 0 to 43 of 51, while median survival stayed within a few seconds of 65 s.

Open the slides

In week 6, median survival read 66.0 s four times running while the runs that killed the first boss went from 0 to 8. Nothing went red. This week names what happened there, and how to catch it next time.

The claim#

A measurement saturates when everything you point it at gets the same score. A saturated measurement still prints numbers, still passes its tests, and still looks like evidence. The only way to catch it is to ask whether it has ever told two things apart.

Three ways the specimen’s instrument saturated#

A ceiling. The first simulated player had no reaction time and, on the game as it was then, won 15 runs out of 15 with every build it was given, including builds that were doing nothing (c34ed0b). Every card looked equally good because nothing could look bad.

A floor. Give the 150 ms player extra damage and refuse every other card: median survival reads 65.3 s, 65.3 s, 67.4 s, 61.3 s, 62.7 s, 62.3 s across six levels, 51 runs each. It looks like one wall. With no damage or one level, no run gets past the first boss. From two levels on, 32 to 43 of the 51 runs kill it, and still end at about the same time. The dial is connected to something the metric cannot see.

A cap. Count the 100 ms player’s kills in the first 38 s instead, median of 51 runs. Ordinary card choices, no cards at all, each of the seven cards banned in turn: 34, 34, 34, 34, 34, 34, 34, 34, 34. The game only spawns about that many enemies in 38 s and every build kills all of them. The metric is measuring the spawn rate.

What was done about it#

The test that compared upgrade cards by survival time had caught three dead cards when it could still tell them apart. Once the boss wall went in, it could not fail any more, so it was deleted, with the reason written down (ccf0e33). A test that cannot fail reads as coverage and is noise.

Reading for weeks 9 and 10#

Stefan Freyr Gudmundsson and colleagues, Human-Like Playtesting with Deep Learning (IEEE CIG, 2018).

The makers of Candy Crush compared two bots for predicting how hard a new level would be: one that searches for good moves, and one trained on real players’ moves. The one that played like a person predicted better and cost less to run. Read it before next week, when you become the person.

What you leave with#

For the metric you have relied on most this semester: its spread between your best and worst build, beside its spread between two different sets of 51 seeds. If the first is not clearly bigger than the second, you have been reading noise.