Centrelink and the problems with Weapons of Math Destruction

RISE OF THE MACHINES: "Humans didn’t issue debt notices unless they had evidence a debt existed. To do so without evidence would be to break the law". Photo: FILE PHOTO

RISE OF THE MACHINES: "Humans didn’t issue debt notices unless they had evidence a debt existed. To do so without evidence would be to break the law". Photo: FILE PHOTO

The most frightening thing about the Centrelink malware debacle is the verve with which the government embraced it.

Malware is software designed to do damage under the cover of providing a service.

Unveiling the automated Centrelink debt recovery system mid-year treasurer Scott Morrison and social services minister Christian Porter promised more “accurate and appropriate income testing”.

They were going to work with the prime minister's Digital Transformation Office to “cut red tape and ensure that mistakes are minimised”.

It's theoretically possible for machines to do complex things better than humans. These days chess-playing programs do it (at least they do it to me) but those programs are exquisitely designed and have goals that are properly specified.

What Morrison and Porter promised was an automated system that would issue Centrelink debt notices “better” than human beings.

Humans did the job extremely well. A former Centrelink worker with 30 years experience says they would “look at start dates for employment that customers had declared, see if it was the same for the employer [using Tax Office records] and roughly work out if it lined up.”

“If it looked as if a person had possibly been overpaid they would write to the customer and ask them to call and tease out where the discrepancy was, and ask for proof, if it was still available, in the form of things such as payslips. If the customer didn't have them and it looked like there was a possibility of an overpayment, they would write to the employer to ask for the information. If evidence was collected that the customer had not declared the income correctly and a debt existed, then the debt calculator would raise the debt in accordance with the legislation and the customer would be written to.”

What's important in this description is the humans charged with applying the law didn't issue debt notices unless they had evidence that a debt existed. To do so without evidence would be to break the law.

But a wrongly-programmed computer need have no such scruples. Even better, its decisions can be presented as objective, hard to overturn.

They are used because they are quick rather than accurate. As an expert said, the primary purpose of a workplace hiring system is “not to find the best employee, but to exclude as many people as possible as cheaply as possible”.

By necessity, they do it unfairly. People who are wise to the systems will mention the right words in job applications to get to the top of the pile. Here, it's the persistent and well-resourced people who get the better of Centrelink. They are unlikely to be the hardest up.

Many of the automated systems are malicious, created to do harm in the guise of providing a service. The formula used by Centrelink produces consistently false estimates of debts by dividing by 26 the annual wages employers report paying in order to overestimate income received during the smaller number of fortnights claimants get benefits.

Humans didn’t issue debt notices unless they had evidence a debt existed. To do so without evidence would be to break the law. 

That isn't to say other software can't be designed to do the job better than humans. It's a worthy aim, one Morrison and Porter apparently thought they had achieved.

The man Turnbull hired to prevent such stuff-ups describes what happened as "cataclysmic". Paul Shetler left the prime minister's Digital Transformation Office in November as the Centrelink debt collection program gathered pace. 

Smartphone
Tablet - Narrow
Tablet - Wide
Desktop