Independent watchdog needed to probe Government's use of AI: law, computer science experts

Sunday, 26 May 2019

An independent watchdog is needed to monitor the use of artificial intelligence by government agencies, according to a new report by University of Otago academics.

An independent watchdog needs to be set up to monitor the use of artificial intelligence by government agencies, a group of academics says.

Corrections, police, Immigration, ACC and other agencies use computer-based statistical tools, known as predictive algorithms, to help them make decisions about individuals and their lives – such as whether an offender should be released from prison, based on their likelihood of reoffending.

In the just-published report Government use of artificial intelligence in New Zealand, a think tank from the University of Otago has warned against allowing the practice to continue unchecked.

The group – made up of experts in law, computer science and philosophy – says while predictive algorithms can improve the efficiency, accuracy and consistency of decision making, computer programmes can make mistakes just like humans.

**READ MORE:

Associate professor Colin Gavaghan, director of the New Zealand Law Foundation Centre for Law and Policy in Emerging Technologies at the University of Otago, says Kiwis have a right to know how and why the Government is using artificial intelligence in its decision making.

* Deepfakes and synthetic media an 'abiding concern' - political scientist

* Deepfakes and synthetic media: The new age of disinformation is growing

* Social media influences New Zealanders' opinions on politics and hurts democracy, study says

* PM Jacinda Ardern's 'Christchurch Call' falls short of the scale of terror online**

In the United States, the Compas algorithm, used in the justice system, has been widely criticised for allegedly producing racially biased outcomes. It is said to overstate the risk of black prisoners reoffending, compared with their white counterparts.

Another predictive algorithm was used by the US youth justice system for years before it was tested for accuracy.

Although there was no evidence of similar issues with the artificial intelligence (AI) technology used in New Zealand, the report's authors argued more oversight was needed to ensure the systems were accurate and unbiased.

'If you're at the sharp end of an important decision, whether it's about an application for citizenship or residency or your kids are taken off you or whether you're released from prison, you should be able to check that decision to make sure it's accurate,' co-author and associate professor of law Colin Gavaghan said.

'You should be able to challenge it if you think it's wrong. [But] you can only do that if you can sort of see how the decision was arrived at.'

Predictive algorithms were programmed to analyse data and draw conclusions using information from previous decisions. Because they were 'trained' using previous decisions made by humans, they could learn human prejudices.

'How can a computer be racist? Well, it can and it can't,' Gavaghan said.

'The computer's not racist but if [the] information it's been trained on [is] informed by institutional racism, then that's going to pop out the other end as well.'

Even when ethnicity was excluded from data sets, the inclusion of other demographic information could lead to racial bias 'creeping back in', he said.

'There are other things that can serve as proxies for race. It could be something like post code – some parts of the country correspond quite closely with ethnicity. But we wouldn't know that unless we keep checking at regular intervals to see if things are starting to go off in a worrying direction.'

Alistair Knott, a co-author and associate professor of computer science, said predictive algorithms could be used to make simple decisions, freeing up time for caseworkers to work on difficult or nuanced cases.

However, he cautioned against government agency staff using predictive algorithms 'like an autopilot'.

They should use the systems to inform their decisions, he said.

'Imagine that you are a judge, for instance, and you consult one of these predictive models that says the offender is at high risk of re-offending, how do you weigh that system protection against what you know yourself?

'The machine hasn't talked to the defendant or heard about the particulars of the case.'

The report recommended all predictive algorithms used by government be included in a public register that citizens could inspect. Documents explaining in clear language how each algorithm worked should also be publicly accessible.

Gavaghan, the director of the New Zealand Law Foundation Centre for Law and Policy in Emerging Technologies, said he and his colleagues had proposed each public agency submit reported about when and why it used predictive algorithms, how each worked and its accuracy, to an independent regulator.

If the watchdog was satisfied with the response it would submit the reports to the public register, but if it had concerns it could request more information.

Having a regulator and public register could save taxpayer money by ensuring agencies were aware of the algorithms used by other ministries and departments and prevent them re-inventing the wheel, Gavaghan said.

Predictive algorithms were great tools when they functioned correctly, he said.

'If decisions have to be made about at-risk children or whatever it might be, let them be accurate ones. We don't want to discourage the use of these things altogether. We just want to make sure we get the best of them.'

The project was funded by the New Zealand Law Foundation.