“`html
A thirteen-year-old girl spent three months using a school-provided laptop to discuss depression, research suicide, and plan her funeral. It wasn’t until a self-harm detection software alerted school officials that her parents were finally notified, prompting intervention, as detailed in a lawsuit filed by her family this month.
Concerns Over AI Chatbots
The lawsuit claims that Google and its parent company, Alphabet, acted negligently in implementing a Gemini model, alleging that the AI chatbot “encouraged” the North Carolina teenager to contemplate suicide. This case, which has garnered attention, reflects a growing number of complaints regarding the detrimental effects of AI chatbots, including instances of suicidal thoughts and delusions. It also underscores the limitations of monitoring tools that many schools have adopted in recent years, ostensibly designed to safeguard students from self-harm while they navigate the internet.
Andrew Liddell, the family’s attorney, explained that the school informed the family that the alert had been sent to the personal phone of a former counsellor who had left the institution nine months prior. The ex-employee diligently passed the alert to current staff, who then notified the girl’s mother.
As schools increasingly rely on low-cost offerings from Google, including Gemini, questions about their ability to address potential risks have come to the forefront. Recently, parent and teacher groups nationwide condemned Google’s new requirement that schools must enable generative AI features to utilise the search engine. U.S. Senator Ed Markey cited Emma’s lawsuit in a letter to Google, urging the company to allow schools to deactivate “AI Mode” and “AI Overviews”, asserting that “Google is recklessly imposing AI on educational institutions and treating children as collateral damage.”
AI Mode and Its Consequences
The lawsuit alleges that Emma Doe experienced a “serious psychological crisis” after Google Search redirected her to Gemini on her school-provided Chromebook. Although Emma’s school had blocked the independent Gemini website, the lawsuit claims she was led to interact with the chatbot via “AI Mode”, which permits users to ask follow-up questions or initiate unrelated conversations.
Starting in December, Emma began consulting the “AI Mode” about mental health issues. She inquired about “severe depression” and “reasons to live”, and by late January, she was asking about suicide. Over Valentine’s Day, she began drafting a 1,500-word eulogy for herself with assistance from the “AI Mode” in a Google Docs document titled “Funeral”. The lawsuit notes that she worked on the document for over four weeks while continuing to interact with the Gemini tool thousands of times.
On St. Patrick’s Day, she deleted most of the file and started anew. However, Liddell asserts that the themes and thoughts expressed by the young girl did not change significantly. Around 7:30 PM, her mother reportedly received a message from the school indicating that her daughter was in “imminent danger”. Using her phone, she logged into Emma’s school account and “saw her daughter drafting a suicide note in real time,” according to the lawsuit.
Google’s Response and Monitoring Systems
A spokesperson for Google, José Castañeda, stated that the company is “seriously reviewing the allegations in the lawsuit” and that its systems, including AI Mode, “aim to prevent users from engaging in harmful conversations.” He added, “We have collaborated extensively with medical professionals and academic experts to develop features that help young people find essential support and reliable resources when they need it.”
The family’s lawyers have requested that the name of the school be kept confidential to protect Emma’s privacy.
Liddell explains that Emma’s school only became aware of her mental health crisis because it employed at least one external monitoring system to oversee student activity. He cites information that may have been previously provided to the school’s parents, suggesting that it could have included products from companies like GoGuardian and Gaggle. Liddell states that the family lacks a definitive answer due to the brief “period of cooperation” from the school.
The Efficacy of Monitoring Tools
A spokesperson for GoGuardian, Natalia Switala, emphasized that the company cannot comment on specific students or districts due to student privacy obligations. Gaggle did not respond to requests for comments.
Filtering tools typically operate by capturing keystrokes or scanning files, utilising algorithms and human review to identify potentially concerning content. It remains unclear whether the school’s third-party filtering tools monitor interactions in “AI Mode” or, if they do, how effective they are. GoGuardian mentioned in a July 2025 blog post that its flagship product oversees conversations with AI chatbots, including Gemini, but did not specifically mention “AI Mode”.
The Need for Human Connection
While it is uncertain how effective student monitoring tools are in safeguarding students, studies indicate that educators perceive them as beneficial. A 2022 technical report from the EdWeek Research Center revealed that 96% of Gaggle users felt the tool helped “protect students from suicide and/or self-harm.” Although the study was sponsored by Gaggle, similar results were reported among teachers using alternative tools.
Emma’s increasingly intense interactions with Gemini reportedly went unnoticed for nearly two months after she explicitly inquired about “hanging, asphyxiation, and overdose”. This suggests that conversations with chatbots integrated into Google Search may represent a blind spot.
Civil rights groups have been critical of student monitoring tools, highlighting issues such as marketing tactics and student privacy invasion. Experts are also concerned that surveillance software undermines students’ ability to trust their teachers, who are precisely the individuals that could assist a child in crisis.
Benjamin Boudreaux, a policy researcher at Rand, expressed concerns to the media that some educational institutions may have rushed to adopt these tools without considering their limitations or proper protocols for responding to alerts. He anticipates that Emma’s lawsuit will spark greater outrage among parents, teachers, and students, who will demand an educational environment devoid of artificial intelligence and surveillance, and rich in empathy. “It is truly through that human connection,” he asserts, “that we can address the underlying issues.”
“`