A national body to monitor and audit automated decision-making in government has been recommended by the Royal Commission into the Robodebt Scheme. The findings of the Royal Commission, which run to over 1000 pages, include major criticisms of the scheme and 57 recommendations for the government and the public sector.

“The Commonwealth should consider legislative reform to introduce a consistent legal framework in which automation in government services can operate,” the report concludes.

“The automation used in [Robodebt] at its outset, removing the human element, was a key factor in the harm it did. [Robodebt] serves as an example of what can go wrong when adequate care and skill are not employed in the design of a project; where frameworks for design are missing or not followed; where concerns are suppressed; and where the ramifications of the use of the technology are ignored.”

In 2015 the Government introduced automated systems that averaged a person's yearly income and applied it against their fortnightly payments from Centrelink, generating many incorrect debt notices until it was scrapped in May 2020. The scheme led to a class action lawsuit, and subsequent Royal Commission.

“While Robodebt was not technically an AI system, the Chapter 17 recommendations may still be relevant to the Australian government's broader (and ongoing) public consultation into AI regulation,” notes Herbert Smith Freehills lawyer Raymond Sun.

“There should have been a human in the loop” was basically the key message from the Report.”

The Royal Commission has recommended that “The Commonwealth should consider legislative reform to introduce a consistent legal framework in which automation in government services can operate.

“Where automated decision-making is implemented:

• there should be a clear path for those affected by decisions to seek review
• departmental websites should contain information advising that automated decision-making is used and explaining in plain language how the process works
• business rules and algorithms should be made available, to enable independent expert scrutiny.”

The Robodebt scheme required data to be transferred from the Department of Human Services to the Australian Taxation Office and then back again.

For each year the scheme operated from 2015 to 2019, DHS created a file that contained details of all DHS recipients who, in a specified year, had received a welfare payment, had an outstanding debt to DHS, or were the partner of a recipient that met one of those two criteria.

The file was provided to the ATO, which would then match each DHS recipient with their corresponding PAYG data held by the ATO. The matched records were then returned to DHS.

Dr Elea Wurth, a partner from Deloitte Risk Advisory, was engaged by the Royal Commission to examine the data-matching process.

Dr Wurth reported that there was a lack of proper governance, controls and risk management measures in place under the Scheme.

The ATO argued that its use of the data collected from DHS, the matching and disclosure of matched data back to DHS under the Scheme were all lawful because of general exceptions provided under the Taxation Administration Act.

However, the Royal Commission had doubts as to the correctness of that proposition.

“ … the Commission considers that there is a serious question as to whether information was lawfully disclosed by the ATO to DHS for the purpose of data matching under the Scheme,” the report notes.

It recommends that “The Commonwealth should seek legal advice on the end-to-end data exchange processes which are currently operating between Services Australia and the ATO to ensure they are lawful.”

Dr Wurth told the Commission that “automation has the potential to increase productivity, efficiency, accuracy, and the cost-effectiveness of service delivery. A trustworthy automated system is a system containing automation that is ethical, lawful and technically robust, coupled with good governance and risk management. To achieve trustworthiness, the system must be designed with human agency at its centre.”

The Royal Commission found that “To date, there has been inconsistency in the legal status of automated decision making in Australian government agencies. Numerous Commonwealth laws have been amended to establish a basis for automated decision making, but these amendments have been piecemeal, across a wide body of legislation, and without the necessary further amendments establishing standards for which decisions should be automated and which should not; and appropriately designed systems for transparency, review and appeal.”

“Section 23 of New Zealand’s Official Information Act 1982 provides a person with a right of access to reasons for a decision made by a public service agency or Minister, including a written statement of the findings on material issues of fact; a reference to the information on which the findings were based; and the reasons for the decision or recommendation.132 The introduction of a legal “right to an explanation” in Australian law, in similar terms to s 23 of the Official Information Act 1982 (NZ), could facilitate the creation of a legislative requirement to design explainable systems. “

Former Department of Human Services (DHS) Secretary Kathyrn Campbell was found to have “been responsible for a department that had established, implemented and maintained an unlawful program”.

Royal Commissioner Catherine Holmes accused Campbell of knowing that legislation would need to change if the policy was going to introduce income averaging and not doing anything about it, and also noted that a Centrelink staffer had tried to warn her of the issues being raised.

“She contended that her failure to eliminate its misleading effect was an ‘oversight’,” Holmes wrote.

“That would be an extraordinary oversight for someone of Ms Campbell’s seniority and experience. The weight of the evidence instead leads to the conclusion that Ms Campbell knew of the misleading effect of the NPP [New Policy Proposal] but chose to stay silent.”

Holmes concluded, “It is remarkable how little interest there seems to have been in ensuring the Scheme’s legality, how rushed, its implementation was, how little thought was given to how it would affect welfare recipients and the lengths to which public servants were prepared to go to oblige ministers on a quest for savings. Truly dismaying was the revelation of dishonesty and collusion to prevent the Scheme’s lack of legal foundation coming to light.”

The full report of the Royal Commission is available HERE.