A metric that cannot fail is noise#
SLOP2875 · Week 9
Swipe, or press →
Week 6 read 66.0 s four times running#
while the runs that killed the first boss went 0, 0, 0, 8.
Nothing went red. This week names what happened there.
The claim#
A measurement saturates when everything you point it at gets the same score.
It still prints numbers. It still passes its tests. It still looks like evidence.
The only way to catch it: has it ever told two things apart?
A ceiling#
The first simulated player had no reaction time.
15 wins out of 15, with every build it was given.
Including builds that were doing nothing.
Every card looked equally good because nothing could look bad.
A floor#
Extra damage, refuse every other card, the 150 ms player, 51 runs each. Median survival across six levels:
65.3 s, 65.3 s, 67.4 s, 61.3 s, 62.7 s, 62.3 s.
Runs that kill the first boss, out of 51: 0, 0, 32, 36, 42, 43. The median did not notice.
A cap#
Kills in the first 38 s, the 100 ms player, median of 51 runs. Nine different builds:
34, 34, 34, 34, 34, 34, 34, 34, 34.
The game spawns about that many enemies in 38 s. Every build kills all of them.
The metric is measuring the spawn rate.
A test that cannot fail is noise#
What was done about it#
The test comparing cards by survival time had caught three dead cards, once.
After the boss wall went in, it could not fail any more.
It was deleted, with the reason written down.
Reading, weeks 9 and 10#
Gudmundsson and colleagues, Human-Like Playtesting with Deep Learning.
Two bots predicting how hard a new Candy Crush level is. The one that played like a person predicted better.
Read it before next week, when you become the person.
Thursday#
For the metric you have relied on most: its spread between your best and worst build,
beside its spread between two different sets of 51 seeds.
If the first is not clearly bigger, you have been reading noise.
Week 9, in full#
The page for this week has the numbers and where they came from.