Archive or Delete?

Keeping everything is not a retention policy.

It is the absence of one.

The cost of keeping

Every version you keep is another candidate for being mistaken for the current one.

Week 1’s filename existed because four files were kept and none was marked.

Decay doesn’t wait

  • Bit rot on failing media
  • A format whose only reader stopped shipping
  • A cited link that now returns a different page
  • A cloud pointer that resolves to nothing
  • Encryption whose key is no longer held

Still in the directory. Still opens.

No longer contains what it did.

A directory listing where every entry looks normal, beside what has actually happened to each: bit rot, a format whose reader stopped shipping, a link that now returns something else, a pointer resolving to nothing, and a lost key.

Migration, not storage

Re-encode into currently readable formats.

Week 10’s conversion damage, accepted deliberately: partial loss now instead of total loss later.

A non-lying system

  • Currency is stated, not inferred
  • History is recorded where the work happens
  • Losses are logged
  • Deletion is a decision with an author

Why tooling didn’t fix it

The four files called final were produced by people with access to version control who didn’t reach for it.

The workaround was cheaper at the moment of the save.

The numbers

Google, across a disk population in the tens of thousands: drive failure is poorly predicted by the signals people assume, and SMART alone will not tell you which drive is about to go.

Harvard Law Review: more than 70% of URLs cited in law journals, and 50% in US Supreme Court opinions, no longer resolve to what was cited.

The citation still looks fine. That is the problem.

Deciding is still human work. A well-built system refuses to pretend it has already happened.

Sources

  • Failure Trends in a Large Disk Drive Population Eduardo Pinheiro, Wolf-Dietrich Weber and Luiz André Barroso, FAST ‘07, 17–29. Field data rather than accelerated ageing: temperature and activity correlate with failure far less than expected, and per-drive prediction stays unreliable. usenix.org
  • Perma: Scoping and Addressing the Problem of Link and Reference Rot in Legal Citations Jonathan Zittrain, Kendra Albert and Lawrence Lessig, 127 Harvard Law Review Forum 176, 2014. Reference rot measured in the one corpus where citation is supposed to be permanent. harvardlawreview.org
  • Ensuring the Longevity of Digital Documents Jeff Rothenberg, Scientific American 272(1), 1995, 42–47. Thirty years on, the argument for migration over storage has not been superseded. scientificamerican.com