Design an ungameable review
Design a performance review system for the team whose packets you argued from. The rest of your group gets a week to beat it. Your mark depends on what it survives, what it does about the rater, and whether you named who pays.
Due: 28 May 2027 · Weight: 40%
Two parts, two deadlines, and a week in between where your group attacks it.
The brief#
Design the review system your week-1 self could not beat. Say what it cost, and who pays.
The team is the twelve engineers on the team page, whose packets you read for Assessment 2 and argued over in Assessment 3; one manager, which is you; one skip-level, Renata Vos; and a company that insists on a distribution. Within that, everything is yours: what gets measured, who gathers the evidence, who decides, what the employee writes and what they cannot, whether there is a calibration meeting at all, and what happens when someone disagrees with their rating.
Timeline#
- Thursday 22 April (week 7). Set. Start the trick table now and add a row each week.
- Friday 14 May (week 10), 5 pm. Optional: a draft of the design to your tutor, in the forum. You get one round of written comment by the Tuesday. Drafts are not marked and do not count against you.
- Monday 24 May (week 12), 9 am. The design is posted in the Assessment 4 thread. A design posted late cannot be attacked and cannot pass.
- Monday to Thursday. Your group reads the designs. You read at least two others from your group and prepare a named trick against each; the people who sat in your calibration room know exactly where a design stands.
- Thursday 27 May, 2 pm. The week 12 stand-up. Designs are pinned up by group and attacked in the room; what was pinned to yours is yours to keep.
- Friday 28 May, 5 pm. The justification is posted.
What to write#
Part one: the design. Two pages at most.
The first page is a front page and has exactly three things on it, under these headings.
- What replaces calibration. Week 1 said more than half of a rating is whoever gives it, and calibration is the company’s fix. If you keep the meeting, say why the three minutes are worth it. If you remove it, say what does its job instead; trusting the manager is not an answer the research allows.
- How a rating is contested. Week 12 says a rating people cannot challenge is one they will game harder. Say how a rated person disputes a result, who hears it, and what can change as a result.
- Who pays. One paragraph, naming the people your design sends the bill to. You will expand it in the justification; here it is the summary.
The second page is the system and the trick table. The system, in as few words as it takes: what is measured, who gathers the evidence, who decides, and what the employee writes. Then a table with eleven rows, one per week from 1 to 11, and four columns: the trick, what it stands on, closed or conceded, and how. A conceded trick, named honestly, scores; a closed trick that the attack week reopens does not.
Part two: the justification. Up to 1,000 words, after the attack week, with these sections.
- What it cost. For each closure in the table, who pays and what they pay. Use the weekly cost lines from the stand-up pages as your list of candidates; if your design sends the bill somewhere the course never named, say so, that is the best kind of answer. Removing ticket counts costs you a measure of throughput. Having peers gather evidence costs their time and introduces their politics. Removing the curve costs you the company’s cooperation. Every closure has a price, and the mark is for knowing who pays it.
- What beat it. What the class pinned to your design on Thursday, which of it landed, and what you would change. This section cannot be empty. A design that nothing beat is a design nobody read.
- What you would ship. The version you would actually run, in one paragraph, trade-offs and all.
Rules#
- Post on time or the piece cannot be attacked, and an unattacked design cannot be marked above 49.
- Attack at least two other designs in the stand-up, each with a trick named by week. That is counted under Survival, because a class that does not attack has nothing to report.
- Draft it however you like, with whatever you like. The class will find where it stands regardless.
What you submit#
Two PDFs in the Assessment 4 thread: SLOP3385-<uid>-design.pdf, two
pages, and SLOP3385-<uid>-justification.pdf, up to 1,000 words, at the
times in the timeline. The policies page says what counts as submitted and
how to get an extension; an extension on the Monday deadline moves you out
of the attack week, so ask early.
Marker’s checklist#
- Was the design posted by Monday 9 am? If not, cap at 49.
- Does the front page answer all three questions, and would the research in weeks 1 and 12 accept the first two?
- Does the trick table have eleven rows, each with what the trick stands on and a closed-or-conceded call with a reason?
- Did the attack week reopen anything marked closed? Each reopening moves that row to conceded without the credit for honesty.
- Does the justification name who pays for every closure, and does at least one bill land on someone the course did not already name?
- Is What beat it specific about what was pinned to the design, and does the student say what they would change?
- Did the student attack two designs in the room, with tricks named by week?
The spec
The brief poses the problem and leaves room for a response. The spec is the fixed part: what a marker checks. Some lines can be checked mechanically; the rest call for judgement.
- the design was posted by 9 am on Monday of week 12 and covers what is measured, who decides, and how evidence is gathered
- the design has a front page with the three answers, and a trick table with a row for every week from 1 to 11
- for every trick in weeks 1 to 11 the table says what it stands on and whether it is closed or conceded, and how
- the design says what does calibration's job of reducing rater bias, or keeps calibration and says why
- the design says how a rated person can contest a result, and who hears it
- the justification names who pays under the design, using the weekly cost lines, and what the design gave up for it
- the justification has a section called What beat it, and it is not empty
How it is marked
| Criterion | Weight |
|---|---|
| Coverage. Every trick from weeks 1 to 11 is placed, and closed or conceded | 25% |
| The rater. What replaces calibration, or why it stays | 20% |
| Contest. How a rating is disputed, and by whom | 15% |
| Who pays. The bill for each closure, using the weekly cost lines | 25% |
| Survival. What the attack week found, reported honestly | 15% |
What each grade looks like
| Grade | Marks | What the marker sees |
|---|---|---|
| HD | 80–100 | Every trick from weeks 1 to 11 is placed and either closed with something it can no longer stand on or conceded by name. The front page answers the rater and contest questions with something the research would accept. The bill for every closure is named and lands on someone specific. The attack week found little, and the justification says exactly what it found. |
| D | 70–79 | Most tricks closed, the rest conceded. Both front-page answers present, one of them thin: a calibration meeting kept with no reason, or an appeal route with nobody to hear it. Costs named for most closures. The attack week's findings reported. |
| CR | 60–69 | The trick table is complete but several closures are assertions. One front-page answer is missing, or amounts to trusting the manager. Costs are listed generically. The attack week's findings are mentioned but not engaged with. |
| P | 50–59 | A design and a justification exist, but the trick table has gaps, the front page is incomplete, or the justification claims trade-offs without saying who pays. |
| N | 0–49 | Claims nobody pays. No front page. Not posted by Monday, so nobody could attack it. Or over the limits by enough that the marker stops reading. |