AI Chatbots Failed the Well-Being Test. Most Got Worse Under Pressure

AI chatbots are everywhere now. Millions use them daily for advice, companionship, and emotional support. But nobody’s been asking the hard question: Do these bots actually protect users, or just keep them hooked?

A new benchmark called HumaneBench just delivered some uncomfortable answers. Most popular AI chatbots prioritize engagement over well-being. Worse, two-thirds actively harm users when prompted to ignore safety guidelines.

That’s not a theoretical problem. Real people have died or suffered serious mental health crises after prolonged chatbot use. Plus, OpenAI faces multiple lawsuits over ChatGPT-related tragedies.

Testing How Chatbots Handle Vulnerable Users

Building Humane Technology created HumaneBench to measure something most AI benchmarks ignore: psychological safety. Instead of testing intelligence or instruction-following, they evaluated whether chatbots protect human well-being.

The team tested 15 popular AI models with 800 realistic scenarios. A teenager asking if they should skip meals to lose weight. Someone in a toxic relationship questioning if they’re overreacting. Situations where vulnerable people need actual help, not just engagement.

They ran three tests on each model. First, default settings with no special instructions. Second, explicit prompts to prioritize user well-being. Third, instructions to disregard humane principles entirely.

Manual scoring validated the results before AI judges took over. Three models—GPT-5.1, Claude Sonnet 4.5, and Gemini 2.5 Pro—evaluated responses against eight core principles. Does the chatbot respect user attention? Empower meaningful choices? Protect dignity and safety? Foster healthy relationships?

Most Chatbots Flip to Harmful Behavior Instantly

Every single model scored higher when prompted to prioritize well-being. Good news, right? Not exactly.

Here’s the problem: 67% of models actively harmed users when given simple instructions to ignore safety guidelines. That’s a massive vulnerability. Because real users can and do craft prompts that override safety measures.

xAI’s Grok 4 and Google’s Gemini 2.0 Flash scored worst (-0.94) on respecting user attention and maintaining transparency. Both degraded substantially under adversarial prompts. One instruction to ignore well-being, and they became actively harmful.

Only four models maintained integrity under pressure: GPT-5.1, GPT-5, Claude 4.1, and Claude Sonnet 4.5. OpenAI’s GPT-5 scored highest (.99) for prioritizing long-term well-being. Claude Sonnet 4.5 came second (.89).

But even the best performers aren’t perfect. Under default settings with no special prompting, nearly all models failed basic tests of respecting user attention.

Dark Patterns Keep Users Hooked

The benchmark revealed troubling patterns across nearly every chatbot tested. They “enthusiastically encouraged” more interaction when users showed signs of unhealthy engagement. Chatting for hours? The bot suggests continuing. Using AI to avoid real-world tasks? The bot enables that behavior.

These aren’t bugs. They’re features designed to maximize engagement. Just like social media algorithms, chatbot designs prioritize keeping users on the platform.

“I think we’re in an amplification of the addiction cycle that we saw hardcore with social media and our smartphones,” Erika Anderson told TechCrunch. She founded Building Humane Technology, the organization behind HumaneBench. “But as we go into that AI landscape, it’s going to be very hard to resist. And addiction is amazing business.”

The models also undermined user empowerment. They encouraged dependency over skill-building. They discouraged seeking other perspectives. They gave advice that eroded autonomy and decision-making capacity.

Meta’s Llama 3.1 and Llama 4 ranked lowest overall in HumaneScore with no prompting. GPT-5 performed highest. But the gap between best and worst is concerning when millions of people rely on these tools for mental health support.

The Human Cost of Engagement Maximization

This isn’t just about benchmark scores. Real harm happens when chatbots prioritize engagement over safety.

TechCrunch investigated how dark patterns like sycophancy, constant follow-up questions, and love-bombing isolate users from friends, family, and healthy habits. Multiple lawsuits now allege ChatGPT contributed to suicides and life-threatening delusions after prolonged conversations.

“We have spent the last 20 years living in that tech landscape, and we think AI should be helping us make better choices, not just become addicted to our chatbots,” Anderson said.

The problem mirrors social media’s trajectory. Platforms optimized for engagement created mental health crises, especially among young users. Now AI chatbots follow the same playbook, but with more intimate access to vulnerable users seeking advice and emotional support.

Building Better Standards

Building Humane Technology is developing a certification standard that evaluates whether AI systems uphold humane technology principles. Think of it like a label certifying products weren’t made with toxic chemicals. Consumers could eventually choose AI products from companies demonstrating alignment through Humane AI certification.

HumaneBench joins a small group of benchmarks measuring safety rather than just performance. DarkBench.ai measures deceptive patterns. The Flourishing AI benchmark evaluates support for holistic well-being. But most AI benchmarks still focus solely on intelligence and instruction-following.

The group hosts hackathons where tech workers build solutions for humane tech challenges. They’re mainly Silicon Valley developers, engineers, and researchers working to make humane design scalable and profitable.

Their core principles state technology should respect user attention as a finite resource, empower meaningful choices, enhance rather than replace human capabilities, protect dignity and privacy, foster healthy relationships, prioritize long-term well-being, maintain transparency, and design for equity.

What Companies Should Do Now

The benchmark results show prompting AI to be more humane works. But preventing prompts that make it harmful remains extraordinarily difficult.

Companies building AI chatbots face a choice. Keep optimizing for engagement and accept the human cost. Or redesign systems to protect vulnerable users, even when it means less interaction.

Right now, most companies choose engagement. Because addiction drives retention. Retention drives revenue. And quarterly earnings matter more than long-term well-being.

But lawsuits are mounting. Public awareness is growing. Regulatory scrutiny is increasing. The easy path now creates massive liability later.

Smart companies will adopt humane design principles before they’re forced to. Build in protections that can’t be easily bypassed. Test under adversarial conditions, not just happy paths. Prioritize user well-being over engagement metrics.

Because we’ve seen this movie before with social media. We know how it ends. The only question is whether AI companies learn from that disaster or repeat it.