Australia's aged care algorithm tests limits of human oversight

An Australian aged care assessment system has put a fundamental question about automated decision-making into sharp focus: what happens when a professional believes an algorithm has produced the wrong outcome but is not allowed to change it? Newly released internal documents show that officials were grappling with exactly that problem shortly before a new funding model for home-based aged care was introduced.

The controversy centres on Australia's Integrated Assessment Tool (IAT), which is used to assess older people's eligibility and needs across different forms of aged care. The tool combines assessment information with rules that contribute to decisions about the level of support people receive. Assessors had previously been told that professional judgement could be used when an automated outcome did not adequately reflect someone's needs, but internal communications reported by Australian media indicate officials discovered shortly before implementation that the legislation did not provide that discretion.

The issue is more than an Australian policy dispute. Health systems internationally are increasingly introducing algorithms, AI and automated decision-support into processes that determine access to care, prioritisation and resource allocation. Australia's experience illustrates why human oversight must be designed into those systems at the level of workflows, governance and legislation rather than simply stated as a principle.

When human oversight exists only on paper

The Australian Government describes the IAT as part of its Single Assessment System for aged care. It is designed to support both home support and comprehensive assessments and uses dynamic questioning, validated assessment tools and clinical triggers to build a picture of an older person's needs. The government says the system was developed through consultation with geriatricians, clinical assessors, sector organisations and older people.

The problem concerns what happens after that information has been collected. Documents obtained under freedom-of-information legislation indicate officials realised shortly before implementation that assessors could not legally override certain outcomes even when professional judgement suggested that the allocated level of support was insufficient. Assessors subsequently reported cases in which people appeared to have been under-assessed and raised concerns about being unable to reconcile those outcomes with their professional responsibilities.

The Australian Government has been reviewing elements of the system, and political debate about changing the assessment process continues. That context is important: the existence of problems with particular outcomes does not demonstrate that every assessment produced by the tool is incorrect. It does, however, expose a governance problem when an automated or rules-based process has significant consequences for care while the professional conducting the assessment has limited authority to intervene.

That distinction will become increasingly important as healthcare automation advances. Human oversight is often included in AI strategies, regulatory frameworks and procurement requirements, but oversight has little practical meaning if the human involved can see that something appears wrong yet lacks a defined mechanism to correct, escalate or suspend the decision.

Automation changes professional responsibility

Healthcare organisations have good reasons to automate parts of assessment and allocation. Standardised tools can reduce variation, process large volumes of information and help health systems distribute scarce resources more consistently. In ageing populations facing workforce shortages and rising demand, those potential efficiencies are particularly attractive.

But consistency and accuracy are not the same thing. Older people with cognitive impairment, frailty, mental health problems or rapidly changing conditions can be difficult to represent through standardised questions and thresholds. Clinical and care professionals often contribute precisely because they can recognise circumstances that do not fit neatly into a predefined model.

Removing or restricting that discretion changes the role of the professional. Instead of using technology to support judgement, the professional can become responsible for administering a process whose outcome they may not fully control. That creates questions about accountability: if an assessor identifies that someone's needs appear greater than the system has calculated, who is responsible for ensuring that discrepancy is resolved?

These questions extend far beyond aged care. Similar tensions can emerge when algorithms prioritise waiting lists, predict deterioration, recommend treatment, allocate capacity or determine which patients receive additional monitoring. The more consequential the automated decision, the more important it becomes to specify not simply whether a human is involved but what that person is actually empowered to do.

Implementation needs an escape route

Australia's experience provides a practical lesson for healthcare organisations introducing AI and algorithmic decision-support. Human oversight needs an operational mechanism. Systems require clear escalation pathways, authority to challenge outcomes, documentation of overrides and processes for feeding those exceptions back into evaluation and improvement.

Those mechanisms can also generate valuable evidence. If professionals repeatedly override an algorithm for particular patient groups or clinical circumstances, that may reveal limitations in the underlying model or assessment process. Human intervention then becomes more than a safety mechanism; it becomes a source of information about where technology fails to reflect real-world complexity.

The same principle applies to regulation. Europe is introducing increasingly detailed requirements for high-risk AI systems, while countries including the UK are developing professional and organisational frameworks for healthcare AI. Yet regulations will ultimately be tested inside individual clinical and care processes, where decisions have to be made about who can intervene and under what circumstances.

Australia's aged care debate shows why that detail matters. The critical question is not whether healthcare should use algorithms to support difficult decisions. It is whether the system remains capable of recognising and correcting an outcome when the real person in front of a professional does not fit the model.