The new frontier of crime: What happens when seeing isn’t believing?
Saturday, 23 August 2025
It all starts with a phone call. Your daughter has been travelling overseas and she’s calling from an unknown number to tell you her phone has been stolen. She’s panicked, her voice breaking as she begs you to transfer her $2000 so she can buy a replacement.
It’s her voice. She speaks just as she always does - the same tone, the same stumbles, even the familiar “ums” and “likes”. But it’s all fake. In fact, it’s a deepfake.
A hacker somewhere in the world has trawled through her social media, worked out which country she’s in and scraped together audio clips from videos available online to run through an artificial intelligence (AI) programme.
The scam itself might only last a few minutes, but the damage - financial, emotional, and legal - can be devastating.
Police, lawyers, and tech experts are warning these moments offer a glimpse into a future where crime is harder to spot, trust is harder to earn, and justice is harder to deliver.
A ‘superpower’ for criminals
In short, AI makes committing crimes easier, cheaper, and more accessible, tech expert Finn Hogan explained.
“At the basic level, AI is a tool that can superpower all of the existing criminal enterprises. You take a basic scam, for example, and suddenly it's so much easier. The cost of producing that scam goes to zero in time and money.
“So AI is essentially supercharging the types of crimes that we’re familiar with, but just using technology we’re not familiar with to carry them out,” Hogan said.
Take the term “deepfake”. Ten years ago, you’d have been hard pushed to find someone who could tell you what a deepfake was. Now, deepfakes are synonymous with taking a person’s voice or image and altering them.
Deepfake technologies can be used for perfectly innocent reasons, such as to generate a video of a disco-dancing American President, but they can just as easily be weaponised for more malign purposes, like financially-motivated scams.
Last year, a Taranaki grandmother was duped out of $224,000 after falling victim to an AI deepfake scam featuring Prime Minister Christopher Luxon.
And the technology is improving fast. Earlier this week, former economist and investment manager Gareth Morgan admitted even he briefly thought an AI-generated video of himself was real.
Where once scam emails from so-called “Nigerian princes' gave themselves away with clunky phrasing and poor grammar, New Zealanders are increasingly being targeted by highly-personalised emotive attacks that are altogether more convincing, Hogan said.
Stephen Kho, the director of offensive security at tech company Avast, said that while AI platforms like Chat GPT might have “guard rails” to prevent them being used to generate scams, other websites would allow you to create deepfake images in just minutes.
“That’s what we’d call ‘dark AI’, where the same technology operates in the dark spaces of the internet. So in lots of ways the crime landscape looks exactly like it did before AI, but now everything is faster, more believable, more sophisticated and more prevalent,” Kho explained.
Among the more nefarious uses of deepfake technology is the creation of AI-generated sexually explicit images designed either to humiliate their victim or to extort money for their removal from public platforms, in a practice known as “sextortion”.
Kho said New Zealand now ranked among the top 20 countries most at risk globally of sextortion, according to data collated by Avast.
The impacts on those victimised are already proving devastating. ACT MP Laura McClure told The Post she had spoken to the school of a 13-year-old girl who tried to kill herself after a deepfake sexualised image of her was circulated among classmates.
At the more extreme end were AI-generated child sexual exploitation images, a growing issue for police around the world, according to Detective Senior Sergeant Kepal Richards.
“Even when imagery doesn’t depict ‘real’ victims, the material adds to an ecosystem that incites and glorifies the sexual abuse and harm of children.
“AI-generated child abuse material can be so realistic that resources are diverted from identifying real-life child victims, placing those children at ongoing risk of harm,” Richards said.
Where once we might have talked about new crime types “reaching New Zealand’s shores”, offenders no longer need to actually be in the country to target Kiwis, chair of the transnational and organised crime ministerial group and lawyer, Steve Symon, said.
“There’s a real disconnect between what the public think organised crime is and what it really is. We’re not talking about your cousin’s dropkick boyfriend driving around on his motorbike selling drugs anymore.
“We’re talking about fraud centres set up in the south-east of Asia where they rent out whole apartment blocks and have teams of fulltime scammers utilising AI technologies to optimise what to them is a commercial business. It’s relentless and on an unprecedented scale,” Symon said.
It’s this scale and speed that should be of particular concern, lawyer Arran Hunt added.
“Back in the day you had individual hackers, who would sit behind their screen and send off phishing emails one by one. Then you started to get people developing software so they only needed to hit a button and the scam was out there.
“Now we have AI that can do that to the nth degree. Now the entire world has access to these tools for whatever purposes they want,” Hunt said.
But for Hunt, the technologies are chipping away at something more fundamental: our ability to trust what we see and hear.
“What happens when seeing isn’t believing?”
A new tool in the policing arsenal?
With criminals quick to exploit rapidly evolving technologies, the pressure is on police to keep pace - something Dr Andrew Chen, chief advisor for technology assurance for NZ Police, is acutely aware of.
Police use technology every day, such as automatic number plate recognition technology. But they only use generative AI for two things - on text with Copilot Chat and with one transcription tool.
“We’re very conscious of the fact that technology is advancing, and we don’t want to be left behind. But we need to make sure we’re on the right side of that trend.”
So while it would be nice to have an algorithm to solve, for example, all retail crime, that’s just not how it works, Chen said.
Perhaps where AI could be of most use would be in streamlining police processes for maximum efficiency.
“It provides a lot of opportunities for police to be more efficient in their work and, if we’re efficient in our work, then that means we’ll be able to address more crime.
“This doesn’t necessarily need to be tied to specific crime types. It could be using AI to read through reports staff have written and ensure they’re classified correctly, for example. That then helps ensure that our statistics and data are accurate and we can make informed decisions about resourcing and deployment.”
Another example might be looking at how technology could help link related incidents, Chen said.
“For example, if you’ve got repeat shoplifters, there might be ways we could use AI to link those cases together if it looks like they might have been perpetrated by the same person or group.”
But whether police could use AI to actually detect crimes was a “little bit trickier”, Chen acknowledged.
“It’s easy to get excited and caught up with the hype of it all, but a lot of what police deal with are assaults or emergency responses and that’s not really stuff that AI can help with. It’s also important that we don’t lose the humanity of policing just because we have new technologies available to us.”
Importantly, NZ Police weren’t doing anything in the realm of predictive policing: “We’re not doing the whole ‘ Minority Report’ thing where we’re using technology to tell us who’s going to do something before they’ve done it”.
“There's a very limited appetite for that type of policing. AI has significant potential for policing, but we have higher ethical standards than the criminal side, so there are always going to be ways criminals are able to use AI that runs ahead of what police can do.”
In his position as chair of the transnational and organised crime ministerial group, Steve Symon, is calling for a centralised “data lake”, where all information held by Government agencies is pooled together.
“At the moment different agencies hold different pieces of the puzzle. The police hold a lot of information, but so do the Ministry of Business, Innovation and Employment, Customs, the Department of Internal Affairs and the Ministry of Social Development.
“So the thought is that we take all of this information and make it into a ‘data lake’, where you could run AI over the top to see if, for example, there had been an increase in drug use shown in the wastewater in Hokitika,” Symon said.
The AI might, for instance, then be able to identify a pattern of large cash deposits into a KiwiBank account on Main St on a Friday afternoon.
“Things like that might be hard to pick up without AI, so it could be a really important investigatory tool for us going forward. If we really want to make New Zealand the safest place on earth, we need to get ahead of the pack.”
Fighting against the use of AI in crime could, Symon said, be likened to creating antibodies for a virus.
“You’ve got a virus that’s destroying people, so you go out, figure out what it’s made of and then take a little bit of it to build up our immunity. It’s that same idea here where we should be using the very tools that have been used against us to defeat the problem.”
What happens when AI takes to the courtroom?
If AI can already fool parents into thinking they’re hearing their own child’s voice, or convince voters that a politician said something they never did, what then happens when the same tricks start turning up in a courtroom?
Might we, for example, see a defendant claim the CCTV evidence being presented in court is doctored? According to Chen, in some ways these are exactly the kinds of conversations that are already happening in courtrooms around the country every day.
“From a prosecution perspective, this is where we rely heavily on the chain of evidence and being able to prove that at no step in the process has evidence been tampered with. We do that all the time with forensic evidence.
“So this is an extension of that, where it’s a defence of ‘police have made this up’ or ‘you’ve framed me’. Ultimately it’s for the judges and the jury to determine if they believe that narrative.”
However, it’s also still within police’s technical capabilities to detect when something has been doctored or altered by tracing back through its meta-data, Chen said.
But what happens when police bring evidence before the courts that has been derived, at least in part, by an AI system such as a language-analysis tool? That, Chen said, was yet to be tested in New Zealand courts.
Where the real problems might begin, Hunt and Hogan agreed, is when AI technologies advance even further than they already have.
“We are getting close to what we call the ‘uncanny valley’, where AI-generated images and videos are so good that we can’t actually detect that they’ve been artificially created or the meta-data has been altered.
“Naturally, we look for flaws to make things seem real. But AI now knows that, and can create flaws in material to make it seem real. At the moment, we have experts who can still tell the difference but give it a couple of years and they might not be able to,” Hunt said.
Similarly, Hogan asked what might happen when AI starts operating as an agentic system of its own - that is, where a human has set the programme into motion but the AI system has then gone on to act on its own.
“Our justice system is set up to catch people committing crimes, but more and more the actual action of performing these crimes is taking place through agentic AI systems. The system has been prompted to take the actions by a human, but then taken on the actions itself.
“I think this raises really interesting questions, firstly about how to catch the actual person who set the AI agent into motion, but also about where the liability and culpability lies.”
One MP’s deepfake protest
But courtrooms aren’t the only battlegrounds facing fresh AI minefields. Parliament, too, has been left to grapple with what evolving technology means for our laws.
Few moments captured that urgency more succinctly than when ACT list MP Laura McClure held up an AI-generated sexually explicit image of herself in the house in May to introduce her new member’s bill.
The bill proposes amending the Harmful Digital Communications Act and the Crimes Act to explicitly include deepfake or synthetically created images that are non-consensual and sexual in nature.
Although there were clunky workarounds that could classify the making of these images of another person illegal, the amendment would make it a clear-cut crime, McClure said.
(Any AI-generated image depicting child sexual exploitation is already a distinct crime under New Zealand law.)
The goal of the stunt was simple - impress upon her colleagues how quick and easy the image was to generate. But while the bold move garnered a flurry of support initially, cross-party willingness to support the bill had proved harder to maintain, McClure said.
Whether the amendment goes far enough would be the next question, she said.
“This isn’t a big overhaul, and I do think we need bigger reforms. If you look around globally, we’ve got places like Australia, Canada, the UK, Denmark and the US making moves in this space. They’ve all introduced legislation in this space in the last 12 months, and that’s because these issues are very real.”
But quite where the line should be drawn remains to be seen, with Denmark proposing the legal protection of every individual’s image, voice and movement.
That, McClure said, would risk ruling out creative freedoms like satire.
“What I do think is unacceptable is for somebody to use your likeness to cause harm - financial or emotional. I definitely don’t think we can afford to be sitting around idly waiting because technology is moving much faster than we are as politicians.”
For now, though, the question remains whether our lawmakers can move fast enough to catch up with a technology that isn’t slowing down for anyone.
How to protect yourself from AI-powered scams
1. Stay aware and sceptical
Stephen Kho, director of offensive security at Avast, says awareness is your first line of defence. Scams are now so convincing that anyone can fall for them, so it’s important to normalise conversations about online fraud and remove the stigma.
2. Limit what you share online
The less personal data scammers can gather about you, the harder it is for them to build convincing attacks. Kho suggests using secondary email addresses for competitions or shopping sign-ups, avoiding unnecessary personal details, and being cautious with social media posts that reveal travel plans, birthdays, or family information.
3. Talk about it
Chief Online Safety Officer for NetSafe, Sean Lyons, says one of the simplest but most effective defences is to pause, breathe, and ask for help. Don’t let scammers rush you into a decision. Share suspicious messages with trusted friends, whānau, or organisations like NetSafe before taking action.
“These are incredibly sophisticated scams and no one should feel shame in falling for one. The key is not to face them alone.”