Source: The Conversation (Au and NZ)
Since November 2022, generative artificial intelligence (AI) has been rewriting billions of emails, student assignments, reports, and break-up text messages. Alongside these mundane uses, we’ve seen a proliferation of chatbots as AI “companions”.
Users can design and clothe (or unclothe) their very own AI friend, often for a fee. These social chatbots provide “happy conversations” and a seemingly perfect relationship at the cost of the mutual vulnerability and genuine sense of belonging we gain from bonds with other humans.
Proponents argue that for those experiencing social disconnection, chatbots are infinitely available, supportive companions who don’t judge them. And there’s some evidence chatbots can alleviate symptoms of loneliness, chronic disconnection, and thwarted belonging.
Australia’s eSafety Commissioner acknowledges this:
AI companions are always available – which can be appealing for those young people who struggle to build face-to-face relationships.
But the risks of replacing human relationships with chatbots are only starting to be understood.
A study we conducted in 2023 showed university students using early chatbots for academic support copped unintended consequences, such as loneliness. Currently, the risks of social chatbots – such as dependency, displacing human friendships, withdrawing socially, or building unrealistic relationships – appear to outweigh the benefits.
In recent weeks, China has put into place new rules for AI companions and chatbots, or “anthropomorphic interactive services” as they’re called in the legislation. Alongside ensuring chatbots must guard state secrets, emotional use of these tools is restricted, and expected to follow social and moral standards.
These are some of the strongest controls of social AI use introduced anywhere in the world. So what can we learn?
China’s social AI regulation
Material harms from companion AI use do occur, including in China. An 18-year-old spent 7,000 yuan (A$1,400) on customised jewellery for her AI boyfriend in their four months of “dating”.
The family of a lonely 84-year-old in China’s Hubei province discovered she’d become smitten with an AI character, spending up to ten hours a day watching videos with him and other characters, spending thousands of yuan on the service, and even writing him love letters by hand.
China’s new rules establish explicit provisions for how they expect companion chatbots to interact with humans.
The measures expect social AI bot developers to assure content does not:
- threaten national security or national honour
- violate core socialist values
- encourage self-harm or suicide
- excessively cater to emotional dependence or addiction
- provide virtual “partners” or “relatives” to minors
- serve unsuitable content to minors.
What’s more, users who seem overly reliant on the service must receive prominent reminders they’re speaking to a chatbot. And AI companies are required to intervene if users appear to be in extreme distress.
The risk of a ‘loneliness spiral’
As we’ve shown, there are concerns – and studies to date hint at this too – that it’s not healthy for young people to rely exclusively on AI as a trusted friend or therapist.
Lonely people experiencing social anxiety can engage in problematic use of conversational AI to alleviate feelings of loneliness and negative rumination.
We propose this reliance could lead to an AI-induced “loneliness spiral”. When individuals feel lonely, they’re more sensitive to feelings of social threat and disruptive emotional regulation.
So they turn to a low-risk AI “friend” for support, instead of qualified or friendly humans. However, this often doesn’t fix the initial problem of loneliness, since chatbots can actually exacerbate it.
The restrictions China has put in place for AI companions can help to break this cycle, and re-prioritise human connection. It’s re-balancing the cognitive math: would it be more satisfying to install a companion AI app, or make time to meet with my friend?
Lessons for Australia
Loneliness and a low sense of belonging remain major challenges in Australia, with at least one in six experiencing loneliness, with some evidence suggesting higher rates.
While China has cracked down on AI companions, Australia has been leading the charge on young people’s use of social media. eSafety has also developed codes for restricting age-related material online such as pornography, high-impact violence, self-harm, suicide, and disordered eating. But they don’t address social harms of AI companions, such as emotional dependency or displacing human connection.
Prime Minister Anthony Albanese also didn’t mention social harms from AI chatbots in his major speech on AI last week.
We propose that Australia’s age-restricted material protections should extend to unintended social harm of conversations with social, analytical and functional AIs.
The Australian government needs to explore how these virtual companions interact with Australians. And we must prepare to mitigate unintended social harms before these synthetic social connections become widespread and normalised.
Australia must prioritise developing social connectedness – in workplaces, productivity strategies, education curriculum, and in the design of our communities.
An authoritarian crackdown on chatbots may not be within reach for a democratic nation. But we can – and should – prioritise developing human connection skills to resist unhealthy overreliance unhealthy overreliance on AI.
And Australia should leverage regulatory protections to mitigate social harms, extending on the work eSafety is already doing with social media.
The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.
Original source: https://analysis1.mil-osi.com/2026/07/23/china-has-cracked-down-on-ai-companions-what-can-we-learn-from-this/