Week 3 Workshop: What the ear cannot hear

How would we test whether repeated re-encoding caused a measured change?

10 March 2027

How would we test whether repeated re-encoding caused a measured change?

Learning target. Design a comparison that isolates one variable — and report a result your measure cannot see as a result, rather than as a failure.

Before the workshop#

Open the week 3 pack and read the-comparison.txt and measure.txt. Ten minutes. Do not open the reveal.

Reading. Jonathan Sterne, MP3: The Meaning of a Format (Duke University Press, 2012), chapter 1, on perceptual coding as a theory about a listener. Available through the library; the lecture states the specific claim used.

Last week, and this week#

Week 2 handed you a feature and asked which explanation it supported. You answered by reasoning about mechanisms.

This week you are handed a claim and the comparison someone drew it from. Reasoning about mechanisms is no longer enough: you have to say whether the comparison could support the claim at all, and if not, what comparison would.

The specimen and the measure#

The same source recording as week 2, and six conditions run on it — an exact digital copy, and five encodings at various bitrates and cycle counts.

The score is a simplified deviation score: for each of eight frequency bands, the absolute difference in band energy from the source, all eight added together. One number per condition, computed by eight additions.

It is worth being clear about what that number is not. The band differences are in decibels, but adding eight of them does not produce a decibel quantity. The score is an invented composite on its own scale, and it is useful only for comparing conditions inside this scenario.

Pairwise differences repeat to within 0.5 points. Run the whole procedure again and recompute the difference between two conditions, and that difference moves by up to half a point. The figure is the repeatability of a difference, not of a single score, because a comparison is what you are reporting. A difference of 0.5 or less is a non-detection.

The values are hypothetical, supplied to practise experimental design. They are arithmetically consistent and were not produced by a real encoder. Every file says so, and nothing here supports a claim about how any real codec behaves.

Investigate — 40 minutes#

Find the problem with the comparison you were given (10 minutes). You have two conditions and a claim drawn from them. Compute both totals. Then look at what is different between the two conditions — not just the number you are being invited to look at.

Write a protocol and a prediction (15 minutes). Using protocol-template.txt, design the comparison that would actually test the claim. Name your independent variable, your outcome measure, your control, and everything you are holding fixed.

Then write down what you expect before you compute anything. This is not ceremony. The whole of band-deviations.csv is available to you, and a prediction written afterwards is not a prediction.

Compute your corrected comparison (15 minutes). The conditions you need are already in the file. Compute the two scores and the difference between them, and compare that difference against the 0.5-point repeatability.

Then run your protocol a second time at a different bitrate. The answer changes. Work out why, and what that does to the original claim.

Evaluate — 30 minutes#

Write the one-page protocol and results table you are depositing, then answer three questions in a paragraph each.

1. What did the original comparison actually measure? Not “it was wrong” — say what its difference is a measurement of.
2. What does your non-detection tell you? One of your corrected comparisons produces a difference the procedure cannot resolve. State what that licenses and what it does not. “We did not detect a difference” and “there is no difference” are not the same claim.
3. What would you have to change to detect it? If a difference exists but sits below your resolution, name one change to the score or the setup that might reveal it — and what that change would cost you.

The analysis is in the pack. Open it after your protocol, your prediction and your three paragraphs.

What you should find#

The comparison you were handed changes two things: the cycle count goes from 1 to 10, and the bitrate goes from 320 to 96. Its 9.5-point difference is there in the numbers, and it is not evidence about cycles, because you cannot say which variable produced it.

Hold the bitrate fixed and the picture changes twice over.

At 128 kbps, ten cycles produce a 3.2-point difference — above the repeatability, so it is detectable in this scenario, and roughly a third of what the confounded comparison suggested.

At 320 kbps, ten cycles produce 0.4 points. That is at or below the 0.5-point repeatability, so it is a non-detection: this procedure gives you no basis for saying the two conditions differ.

So the answer inside this scenario is: whether a difference is detectable depends on the bitrate, and at 320 kbps it was not. That is a worse headline and a better finding — and it is a finding about this scenario, not about MP3.

Notice what the exact-copy control does here, as it did in week 1. Ten copy operations, no encoder, a score of zero. That is consistent with the procedure not generating differences on its own. It does not prove the procedure is sound; a control that had come back non-zero would have shown it was not.

Deposit#

Your protocol and results table go into the audio collection as a controlled comparison, with the condition settings recorded alongside the numbers. A result without its settings is not reusable, and week 10 will come back to this collection needing exactly those settings.

The five-minute check#

A comparison changes both the bitrate and the number of encoding cycles. Identify the confound, specify one corrected comparison, and state its remaining limit.

Assessment#

The Chain asks for ten copying steps, a control, and a prediction recorded before you start. Its second criterion is measurement and controls, at 30%: whether your measure suits your question and your control isolates a competing explanation.

Take the protocol template with you. The most common way that assessment goes wrong is a chain that varies two things at once, and then a conclusion that names only one of them.