Concerns growing critical modelling software used by regional councils is flawed

Wednesday, 23 May 2018

A modelling system used all over the country to form key decisions around water use may be seriously flawed.

Scientists and mathematicians say Overseer, the modelling tool being widely used by regional councils to inform water management policies, was never designed to be used that way.

Overseer is web-based software tool, jointly owned by the Ministry for Primary Industries, AgResearch and the Fertiliser Association of New Zealand, that helps farmers and growers make better environmental outcomes.

It receives over $1.7 million a year in taxpayer funding.

**READ MORE:

* [Te Waikoropupū Springs hearing adjourned over challenge to evidence

* Soil scientist questions accuracy of Overseer modelling

* There's a better way for councils to use Overseer](https://www.stuff.co.nz/business/farming/opinion/66243959/theres-a-better-way-for-councils-to-use-overseer)**

It estimates how a farm is using available nutrients, what and how much fertiliser is needed to maintain soil fertility, and whether damaging nutrients are leaching into the waterways.

Dairy NZ says Overseer is the most effective modelling software that is available at this time.

But Overseer itself admits to a level of uncertainty, saying its results could never be 100 per cent accurate.

More and more regional councils are adopting it as a tool to manage and improve their waterways, but soil scientist Doug Edmeades believes it was never designed as a regulatory tool.

'It was designed to make decisions about what factors a farmer can use to control or reduce the levels of nitrate in the soil.'

Overseer was a qualitative tool – not quantitative, he said.

'One must be very careful about interpreting the numbers for those reasons.'

Former AgResearch scientist Dr Samuel Dennis doesn't trust the numbers produced by Overseer, so he's designing a new technology he hopes will give much more accurate results.

He said it was really good as a management tool for fertiliser recommendations and making better environmental decisions, but its numbers were based on estimates.

They were just 'not sound enough' to use as a compliance tool, he said.

It was recently brought into the spotlight after mathematician Dr John Gamlen criticised a key Tasman District Council scientist Andrew Fenemor at a hearing involving special protection for Te Waikoropupu Springs.

Gamlen called Fenemor's nitrate modelling at the springs 'flawed', resulting in wrong conclusions.

Fenemor, a senior scientist at Landcare Research, presented expert evidence about how nitrates would affect the springs if further irrigation was allowed.

Fenemor used Overseer to conclude it would not harm the springs if irrigation was doubled in the catchment.

But like Gamlen, Professor Graeme Wake also believes Overseer is 'flawed' and has significant mathematical fallacies.

The professional systems modeller and emeritus professor of industrial mathematics at Massey University Auckland said many had been concerned for some time about Overseer.

'It was a good idea but it has taken up the slack by doing modelling, which could be done a lot better with more correct algorithmic procedures.'

Wake said its flaws were easy to spot for mathematicians and professionals.

The model had been changed a lot over the years, with some serious errors introduced and fallacies committed.

Most significantly, Overseer did not recognise that things could evolve from a static situation into a dynamic one.

'This is why this model fails…and users try to fix it by introducing fudge factors which works for a while until circumstances change.'

Wake said no mathematician had ever been allowed to look at how Overseer worked, despite asking many times.

There was basis for a legal challenge based on the model's mistakes.

A water resource engineer and scientist, Dr Aroon Parshotam, called it a 'ridiculous' situation.

Overseer was a 'black box' model, meaning its developer had never publicly disclosed how it was made.

Parshotam questioned how a single individual could develop a model in-house that was being used for regulatory purposes nationally, and no one knew how it was made.

He called it a 'double-edged sword'. If they were to release the formulation, anyone would be able to come up with a similar model.

'But that is not how successful science works. It has to be out in the open and debated,' he said.

If it was challenged in court, Parshotam said Overseer would not survive.

'It's not transparent; it's not open, and people are making a lot of money from it.'

Overseer chief executive Caroline Read called it a 'sophisticated science model', integrating over 30 years of applied farm science to model really complex biophysical processes.

But it was a model – not a measure, she said.

Environmental models were usually a series of mathematical equations that attempted to mimic real life, but their results could never be 100 per cent accurate.

'We calibrate Overseer modelling against real measured data where it's available to improve the certainty of the model's estimates, and we also peer review and test the science behind the model,' she said.

But one of the challenges of evaluating the model was that it needed good evaluation data, and they simply didn't yet have the quality of data sets for all farm types in all conditions.

When used well, Overseer was a really powerful tool for helping farmers improve their farm's environmental sustainability.