Week 4: One run tells you nothing

The same player on the same game ends anywhere between 29.8 s and 120 s. This week is about how many runs a claim needs before it is worth saying out loud.

16 March 2027

Is it me or is it the game? A single death is one draw from the dice. One run said 69.3 s; the median of fifteen said 81.5 s, and nothing had changed but the seeds.

Open the slides

Seed 97 ends at 69.3 s. That sentence is true, and it is an anecdote.

The claim#

A single run is one draw from a distribution, and the distribution is the thing you are trying to talk about. Quoting one run, or the best run, or the run you happened to watch, is how balance arguments go wrong while everybody in them is being honest.

How wide it is#

The 150 ms player from last week, ordinary upgrade choices, 501 different seeds. The shortest run ends at 29.8 s. The longest reaches the full 120 s. Nothing differs between them except the dice.

Here is the median as the number of runs grows:

Fifteen runs put the median 12 s above what one run said. Five hundred and one pull it back down by 8 s. It never stops moving completely. What you get to decide is how much movement the decision in front of you can tolerate. Run the same count again on different seeds and the median moves again; how far it moves is the smallest difference that count lets you claim.

This course uses 51 as its working number. It is odd, so the median is a run that actually happened, and 51 runs of the specimen cost 166 ms. There is no excuse for quoting fewer.

What you leave with#

The run count you will quote from now on, and the smallest difference you will claim at that count: how far a second set of seeds moves your median.

Run it yourself