Making online platforms safer by design
Friday, 12 September 2025
Jacqui Dillon is executive director of Amesty International Aotearoa; Anjum Rahman is founder and facilitator of Inclusive Aotearoa Collective Tāhono. The two organisations have launched the #noharmware campaign.
OPINION: This week the Education and Workforce Select Committee heard from people as part of its inquiry into online harm and young people. Many submitters called on decision-makers to take action to address the root causes of harm. We hope they listen.
The rise of the internet has opened up incredible possibilities, allowing people to connect instantly and globally. However, without proper regulations, we've witnessed the creation of digital platforms that foster deeply harmful and toxic online spaces.
It might feel possible to isolate online harm as being in another world – the online world, where you can log off or close your accounts if it gets too bad. The problem is, there isn’t really a separation between the online and offline, it’s more like a continuum. What this means is that harm occurring online can hurt you and those around you even if you’ve never stepped into the world of X, Meta, gaming platforms or other online spaces.
At the most basic level, many online platforms work through algorithms, sets of instructions or steps to solve problems. Recommender algorithms suggest content. Reaction buttons are learning what elicits a response from you. Even if you give no reaction, platforms are learning from the time spent looking at posts. Internet searches learn from the sites you click and read, and will, over time, serve you content they think aligns with your views and interests.
On one hand, this can be helpful as you mainly get to see what you’re interested in. The downside is that these filters can seriously narrow the perspectives you see.
This takes us to a more insidious harm. The link between online platforms and radicalisation and violent extremism is well known. Numerous examples show what happens when hate fuelled online spreads into offline violence.
Recent Amnesty research showed how X, formerly Twitter, played a central role in the spread of harmful content which contributed to racist violence against Muslim and migrant communities in the UK in 2024.
Another report released this month analysed how X contributed to the spread of hate and harassment against members of the rainbow community in Poland, allowing for the targeting of the community to become increasingly normalised.
In 2017, the Rohingya faced utterly horrific violence as part of the Myanmar security forces’ campaign of ethnic cleansing. In the months and years leading up to the atrocities, Facebook’s algorithms were intensifying a storm of hatred against the Rohingya which contributed to real-world violence. The UN’s Independent International Fact-Finding Mission on Myanmar ultimately concluded that the “role of social media [was] significant” in the atrocities that ensued.
There are many other examples that illustrate the gravity of harm that needs action. And without regulation Aotearoa is not immune to the same harm.
Before jumping to a solution we need to identify the cause. A core part of the problem is that we’re seeing online platforms designed to promote high engagement content (regardless of the harm). This is driven by a business model where harmful content drives engagement and therefore revenue.
Sensationalism in mass media is not new, but social media goes well beyond the old adage “if it bleeds, it leads”. It can systematically privilege extreme content including conspiracy theories, misogyny, and racism to keep people on platforms for as long as possible.
At the same time we’re hearing concerns about unjust censorship, with reports detailing accounts being taken down unfairly and the use of shadow-banning (or algorithmic suppression), where content is less visible for certain topics. An example is content related to the Gaza genocide.
Most people would agree action is needed to address the harm. The Government is considering a proposal to ban social media for young people, but this is wholly inadequate in the face of the gravity and scope of harm and doesn’t address the root causes. It puts the burden on young people to change, rather than the big tech companies who are responsible. This isn’t fair.
The good news is there are overseas examples of what can be done to make a real difference. It’s about the creation of laws that place obligations on tech companies to make their product safer by design. We need clear requirements for transparency and accountability, just as we expect for other products we use. What this looks like in practice is an independent regulator, with requirements for companies to identify risks and show how they’re addressing them. It’s about requiring tech companies to be transparent about their algorithms and how they work.
We can learn from overseas experience in Australia, the UK and the European Union, who have already adopted such laws. While there are some issues with such legislation, using a strong human rights-based approach, along with independence from political interference, should help us avoid many of the pitfalls.
Legislation and regulation must actively recognise the core drivers behind harm and put safeguards in place to transform online spaces into a more positive force. Our #noharmware campaign calls on the Government to enact laws to help make online platforms safer by design.