Big Story
Averaging towards the mean
At an AI in the NDIS summit panel, the concern that came up most wasn't privacy, and it wasn't cybersecurity. It was simpler than that: as automated tools start drafting plan recommendations, they tend to "average towards the mean" — producing a plausible plan for a typical participant, when no participant is actually typical.
The panel — EY's Christina Larkin, former NDIA deputy CEO Anthony Vella, UTS's Darren Chua, and Jessica Martin of Bella Sláinte — pushed past the usual "human in the loop" reassurance. Their sharper question: when a model generates a budget, can the delegate who reviews it actually change it, or can they only ask the model to try again with different inputs? Nominal human oversight without real authority to override is exactly the failure mode the Robodebt scheme was built on. The panel's proposed standard was blunter than "human in the loop" — they called it a "meaningful human decision."
Whose data trains the model
Historical bias in the underlying data compounds the problem. One panellist pointed to a pattern well documented in health research generally: bias in health data tends to skew against women, because early medical research disproportionately studied male subjects. A planning model trained on that history doesn't correct the skew — it inherits it.
What changes in December
Privacy Act changes taking effect on 10 December 2026 will require disclosure whenever an automated system substantially informs a decision about access to services — a description that fits NDIS planning directly. One assurance specialist on the panel was candid that auditing a genuinely "black box" system for bias is hard to do properly, which raises the stakes on getting the disclosure right rather than treating it as a formality.
The NDIS Quality and Safeguards Commission is already building its own model in this space: an "intelligent risk engine," part of a $160 million program, intended to assess risk across participants, providers and workers alike. Not every voice on the panel was opposed to automation — one speaker made the case that it can cut administrative burden and enable continuous monitoring across an entire participant population, rather than the sampling that's standard today.
What the panel wants done before December
- Publish the override rules for any model-assisted decision.
- Disclose where automated systems substantially inform planning decisions, ahead of the Privacy Act deadline.
- Commission independent bias audits, not internal sign-off.
- Guarantee a real human review pathway — one that can change the outcome, not just re-run the model.