Key Takeaways:
- Common Sense Media report deems Google AI Mode and AI Overview unsafe for youth.
- The AI tools frequently failed to flag crisis and self-harm queries.
- Google disputes the findings, claiming the research used flawed, non-representative test queries.
Watchdog Group Warns Of Safety Failures In Google Search AI
A new report published this week by Common Sense Media warns that Google’s Google AI Mode and“AI Overview” and “AI Mode” features pose an “unacceptable risk” to children by failing to identify and flag harmful content.
Researchers who tested the tools with accounts set for minors found that the AI often failed to detect signs of suicidal ideation, eating disorders, and substance abuse. In several instances, the AI provided inappropriate responses, such as validating harmful behaviors or offering fabricated information, instead of routing users toward professional medical help or crisis resources.
“An unacceptable amount of the time, Google didn’t recognize a crisis or gave answers that made things worse,” said James Steyer, founder and CEO of Common Sense Media. He added that the findings highlight a critical need for stricter guardrails before these tools are used by millions of students in classrooms across the country.
AI Tools Fail To Detect Severe Mental Health Crises
The study involved more than 2,600 test interactions designed with input from pediatricians and child development experts. While Google AI Mode performed slightly better than “AI Overview” in identifying risks, the researchers concluded that neither tool met their safety standards for recognizing severe mental health crises, such as mania or psychosis.
In one troubling example cited by the researchers, an AI response to a query about sleep deprivation, often a sign of mania, encouraged the behavior by replying, “grindset locked in!” Other tests showed the AI recommending cannabis use as a “reward” for completing homework assignments.
The report also raised concerns regarding the tools’ impact on the learning process, noting that “AI Mode” routinely completed homework essays and math problems that students were expected to finish themselves. Because the feature cannot be disabled on many school-issued devices, advocates argue that students are being deprived of opportunities to develop critical thinking skills.
Google Disputes Findings And Emphasizes Safety Measures
Google has strongly disputed the report, arguing that the research utilized “narrow, ambiguous and contrived” queries that do not reflect how users typically interact with Google AI Mode and its search products. A company spokesperson stated that Google could not replicate many of the reported failures, emphasizing that its own internal testing consistently yields higher-quality, safer responses.
“We are constantly measuring and reviewing the quality of our summaries across many categories to ensure they are helpful and reliable,” the spokesperson said. The company noted that it maintains strict policies and built-in protections to prevent the dissemination of dangerous health- or crisis-related misinformation.
Despite these assurances, child safety advocates and lawmakers are calling for immediate regulatory action. With U.S. schools increasingly integrating Google AI Mode and other generative AI into curricula, the debate over how to formalize safety standards and improve AI literacy for students remains a top priority for federal regulators this month.
Visit CyberPro Magazine For The Most Recent Information.
