Week 11 Lab: Forty Things at Once
This week
Track enough about yourself and something will always correlate with something, which is how a dashboard lies to you with true numbers.
Before you run this
- When to stop
- Stop and hand the logging to us if the tracking becomes the point, if you are reluctant to end the week, if a missing entry bothers you, or if you find yourself adjusting your day to make a number look better.
- What this is not
- This week deliberately induces over-measuring in order to show you what it feels like, and it lasts four days. It is not a template. Anyone for whom detailed self-tracking has previously become compulsive should take the alternative without discussing it with anyone.
- If this one isn't for you
- Use the class's synthetic dataset, which is generated from pure noise. It produces the same spurious correlations and asks nothing of your own day.
What you bring to the lab
The lesson below is how you get there. The lab is where you show it.
- you log at least twenty variables for four days
- you find a correlation you would have believed, and say why you would have
- you can state how many comparisons your dataset makes available
Log twenty things about yourself for four days. Thirty is better. Sleep, mood, steps, coffee, water, screen time, how many people you spoke to, what you ate, the weather, whether you wore the good jacket.
Then go looking for patterns. You will find them. They will be real numbers, correctly calculated, and mostly meaningless.
The arithmetic of finding things#
Twenty variables give you a hundred and ninety possible pairs. At four days each, some of those pairs will line up beautifully by chance alone, not because you made an error, but because that is what chance does at that many opportunities.
Multiple comparisons is the name for this problem, and its signature is a specific kind of story: I discovered that I sleep better on days I drink more water. The number behind that sentence can be impeccable. The sentence is still probably about nothing, because it was chosen from a hundred and eighty-nine alternatives after the fact.
This is week 6’s lesson arriving from a different direction. There, the danger was deciding what you were testing after seeing the result. Here, the danger is testing everything and reporting the winner.
The salmon#
In 2009 a group of neuroscientists put a dead Atlantic salmon into an fMRI scanner and showed it photographs of people in social situations, asking it to determine what emotion the people were experiencing. They then analysed the scan the way many papers did at the time.
The salmon showed significant brain activity.
The salmon was dead and on a fishmonger’s slab hours earlier. The point of the exercise was that a brain scan contains tens of thousands of voxels, each one a separate test, and if you do not correct for how many questions you asked, you will find answers in a fish.
What this does to your own dashboard#
Every self-tracking application you have ever used does this by design. It shows you the correlation it found, not the two hundred it looked at. It cannot tell you how many comparisons went into the one insight it surfaced, and usually it does not know.
So the assessable part of this week is not a finding. It is a number: how many comparisons did your dataset make available? Once you can say that out loud, the insights get much quieter, and much more trustworthy.
Evidence on file
- 1 salmon blinding not possible
A dead salmon, scanned while shown photographs of people in social situations, produced 16 significant voxels out of 8,064 under uncorrected statistics. Correcting for the number of comparisons left none.
Bennett CM, Baird AA, Miller MB, Wolford GL, Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon, poster, Organization for Human Brain Mapping 2009- History, not measured
A group of neuroscientists scanned a dead Atlantic salmon in an fMRI machine while showing it photographs of human social situations, and recovered apparently significant brain activity, presented as an argument for correcting when you test many voxels at once.
Bennett CM, Baird AA, Miller MB, Wolford GL, Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon, poster, Organization for Human Brain Mapping 2009; Journal of Serendipitous and Unexpected Results 2010
Teaching team
- Dr Wren Halliday, convenor
- Priya Ramanathan, tutor