Step forward for Kiwis' rights when decisions made by computer

Tuesday, 28 July 2020

Agencies will be free to decide themselves whether decisions are important enough to be covered by the Government’s “algorithm” charter.

ANALYSIS The Government has developed a charter that is designed to protect people from bad decisions made by computers – or more accurately from bad decisions by the people who program them.

The charter, released by Statistics Minister James Shaw, and so far signed by 21 government agencies is designed to ensure computer “algorithms” are being used in a fair, ethical, and transparent way in the public sector.

Fledgling attempts to use algorithms and “machine learning” to improve government decision-making have resulted a few scandals in the past.

The Social Development Ministry got itself in a tangle in 2015 when then minister Anne Tolley abruptly canned a controversial experiment to predict children at risk of abuse.

* Independent watchdog needed to probe Government's use of AI: law, computer science experts

The Department of Corrections is among agencies that have agreed to abide by the new algorithm “charter”.

* Algorithms are everywhere but the public sector seems scared to use them

* Government agencies are using algorithms to help make decisions

Computer-assisted decision-making also hit the headlines for the wrong reasons in 2017 when an Immigration NZ official appeared to suggest computer algorithms were being used to prioritise the deportation of overstayers based on factors such as race.

That proved to be more of a false scare.

But a government stocktake in 2018 found computer algorithms were being used by agencies to help make a wide range of potentially life-changing decisions about the treatment of Kiwis.

Police have used software to help predict whether people with a history of family violence are likely to commit crimes against their victims within two years.

The Corrections Department was an early pioneer and has long used software to forecast the likelihood of inmates re-offending, when they come up for parole hearings.

Computer forecasts have helped determine how vigorously visa applicants are screened and whether people are offered automatic tax refunds by Inland Revenue.

The Government’s claim on Tuesday that it had become “the first in the world to outline a set of standards to guide the use of algorithms by public agencies” is perhaps a bit of a stretch.

In theory at least, the General Data Protection Regulation adopted by the EU in 2018 gives Europeans a general right – not limited to government agencies – to an explanation when automated decisions are made about them.

The EU rule also gives people the right to have a human involved in any 'significant' decision affecting them, unless there are suitable legal safeguards.

New Zealand’s charter doesn’t appear to give that explicit right, requiring only that agencies “provide a channel for challenging or appealing decisions informed by algorithms”.

But more laudable is its focus on making sure people can find out what is going on when important decisions are made by computer.

Agencies will need to clearly explain how decisions are informed by algorithms, and make the information about the “data and processes” available.

They will also need to nominate a point of contact for public inquiries about algorithms.

The charter requires that algorithms are regularly peer-reviewed to spot “unintended consequences”.

The focus on transparency is vital, because when it comes to computer-assisted decision-making, the problem is that “you don’t know what you don’t know”, and the pitfalls are in the detail.

Precisely what variables is Corrections looking at when deciding whether a prisoner might be at risk of reoffending, and what processes have been implemented to avoid bad or biased algorithms, and biases in the underlying data that they are crunching?

If rigorously implemented by agencies, the Government charter should provide an easy mechanism to get answers to those sorts of questions for the first time.

The biggest fudge is that computerised decisions have been divided up into three categories, based on a subjective assessment by agencies themselves of their frequency and “risk”, for which the charter’s terms must, “should” or need not apply.

That aside, the approach looks positive.

A problem with the EU “right to appeal”, on the other hand, is that it is of little use if you have no inkling a decision has been made by a computer in the first place.