Week 12 Lab: The Day You Don't Measure
This week
When to stop experimenting, why every answer you found has an expiry date, and what you are left holding.
What you bring to the lab
The lesson below is how you get there. The lab is where you show it.
- you nominate one measurement you will keep and say for how long
- you nominate one you are stopping, and why
- you can state the expiry date of your own main finding
This week you do not measure anything.
That is the exercise, and for a certain kind of student it is the hardest one in the course.
The course cannot give you an answer#
You came in wanting to know when to sleep, whether to eat breakfast, how long to work. Twelve weeks on, the honest position is this: you have found some things out about yourself, and every one of them has an expiry date, because week 10 showed you that the subject keeps moving.
That is not a disappointing result. It is the result. A course that sent you away with a list would be handing you the one thing this subject cannot support: a fixed answer, drawn from one person across one stretch of days, and measured in March. The published guidance has the same shape and is honest about it, which is why it comes as ranges and aims.
What survives is not the answer. It is that you now know how to ask again.
Stopping is a skill#
Everything in this course had a stopping rule, and by now you should have noticed that the stopping rules were the most useful writing on each page. They were there for your safety, but they were also doing methodological work: a measurement without a planned ending quietly becomes a habit, and a habit is not an experiment. It is just something you do now.
So: pick one measurement to keep, and say for how long. Pick one to stop today. Both choices are assessed, and the second is worth more, because it is the one nobody does voluntarily.
Why registries exist#
By the early 2000s medical journal editors had a problem they could no longer ignore. Trials were reporting outcomes that had been chosen after the results came in, and trials with unwelcome results were not being published at all, so the literature was a record of what had worked rather than what had been tried.
The fix was procedural, not statistical: register the trial, with its outcomes, in public, before enrolling anybody. From mid-2005 the major journals would not publish anything that had not done so.
You did the personal version of this in week 6. It is the same instrument at a different scale, and it exists for the same reason, not because researchers are dishonest, but because nobody, including you, can be trusted to remember what they expected once they know what happened.
The last hour#
We spend it on the day you did not measure. What did you notice without writing anything down? What did you miss?
Some of you will find the unmeasured day a relief and some of you will find it unnerving, and both reactions are worth saying out loud, because they tell you something about what the measuring was for.
Evidence on file
- 323 trials blinding not possible
Among trials that had been registered properly, 31% published a different primary outcome from the one they registered. Where the direction could be judged, the change favoured a statistically significant result 83% of the time.
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P, Comparison of registered and published primary outcomes in randomized controlled trials, JAMA 2009;302(9):977-984- History, not measured
Medical journal editors began requiring that trials be registered before enrolment, from mid-2005, because outcomes were being selected after results were known and unfavourable trials were disappearing entirely.
De Angelis C et al., Clinical Trial Registration, a Statement from the International Committee of Medical Journal Editors, New England Journal of Medicine 2004;351:1250-1251
Teaching team
- Tobias Veck, tutor