Week 8 Sim Session: Rating the samples

You apply the ICAO descriptors to recorded speakers, discover you disagree with each other, and then find out what the examiner gave them

28 April 2027

Today you do the examiner’s job badly, which is the fastest way to understand what it involves.

Before you arrive#

Read the ICAO rating scale descriptors for levels 3, 4 and 5 across all six criteria. Bring them. The session is unusable without the descriptors in front of you.

In the session#

Calibration, fifteen minutes. Two samples, rated as a room, out loud, with Priya arguing the descriptors. Nobody is right yet; the point is to fix what the words in the scale mean before using them.

Solo rating, forty minutes. Six samples, rated silently and individually. All six criteria for each. Write the criterion that decided your overall rating.

Samples are real, anonymised, and used with permission for training. Some are non-native speakers, some are native. Do not assume the difficulty runs the way you expect — one of the six is a native speaker whose local shorthand and speed make comprehension harder than anything else on the tape, and the room regularly rates that one highest.

The spread, twenty-five minutes. Ratings go on the board anonymously. There will be a spread, usually two levels wide, sometimes three.

We then look at which criterion drove the disagreement. Almost always it is comprehension or interactions — the two that depend on the unexpected — and almost never pronunciation. That is the week’s finding: the criteria that matter most are the ones a booked interview measures least reliably.

What the examiner gave them, ten minutes. Priya reveals the operational rating each sample actually received. It will not match your consensus on at least two of them, and she will explain the reasoning without defending it as obviously correct.

What we are careful about#

The samples are people’s voices and people’s licences. Rate the speech against the descriptors. Do not speculate about nationality, and do not use the samples outside the session.

If you find yourself rating an accent, stop and find the descriptor you think it fails. Usually there isn’t one, which is the point.

Afterwards#

Bring your disagreement to the audit in week 11 if it is relevant — several past audits have proposed rating scales for their own domain, and the honest ones cite this session as the reason they included a limits section about assessor variance.