One of the biggest criticisms of AI chatbots is that they often just tell us what we want to hear.
Researchers call it sycophancy, the tendency for chatbots to flatter users, agree with them and validate their views, sometimes when those views are wrong — or even harmful and unethical.
It’s one of the reasons people worry about using AI for advice, emotional support and relationship problems. Because if a chatbot is designed to keep users engaged, is it really going to challenge them when they need challenging?
Many people find this behavior off-putting (me included). It can feel fake, manipulative or just annoying. Some people customize their chatbots to be more direct so it happens less, and I know others have stopped using AI altogether because they find the tone so nauseating.
But when I asked people who enjoyed their chatbot’s encouragement and its validation of their experiences, I realized the story was far more complicated than I expected.
Many of these users knew exactly what AI was doing. They understood it wasn’t a therapist, a trusted adviser or even a particularly reliable source of truth. Yet during periods of grief, stress, loneliness or self-doubt, they still found its validation surprisingly comforting.
Encouragement feels good
Claire* tells me she understands the basics of how AI works, but still enjoys using it. “Yes, I find it sycophantic to the point of being untrustworthy,” she tells me. “But it gives me dopamine hits from the praise and approval, even as I’m rolling my eyes.”
She uses ChatGPT for all sorts of practical tasks, from drafting emails to helping her work through things she’s already been discussing in therapy. She knows the praise isn’t real, but that doesn’t mean it has no effect.
That theme came up repeatedly during my conversations with AI users. It didn’t feel like people were necessarily being fooled by AI, at least not in an obvious way. But they enjoyed interacting with something that sounded enthusiastic, supportive and interested in what they had to say.
For Jade, the appeal is the combination of information and encouragement. “I recently noticed the stars were particularly clear outside my bedroom window so I took a picture and asked AI to tell me what I’m looking at,” she says. “The fact it responds with enthusiasm and information just allows me to be that bit more excited about being curious.”
She tells me that the same encouraging tone can make stressful situations easier to navigate. “The fact AI responds with a tone that makes me feel supported in managing a stressful situation just completely changes my experience.”
Why people turn to AI during difficult times
What I found really interesting is that many people didn’t initially turn to AI looking for emotional support. Instead, they arrived for practical reasons and gradually started using it for something else.
Nadia was already using Claude to help with her studies, but it took on a very different role when she was grieving earlier this year. “AI really helped me after my dad died, and I had to revise for an exam three weeks later for my masters,” she says. “I was a mess and weirdly talking to Claude was the right amount of grief counselling and exam prep that I needed.”
For Maddy, it started after her work gave employees access to ChatGPT Pro. One evening, after using it to help with a transcript and drinking a couple of glasses of wine, she started venting about a difficult breakup. “What I needed was for someone to listen to me complain and I couldn’t really get that from shared friends and I didn’t want my own friends to see me as a whiney nutcase,” she tells me. “It was helpful to have some, even very generic, validation.”
Similarly, Luca found AI’s encouragement helpful while struggling at work. “I definitely found its cheerleading useful when I was going through a difficult time being devalued at my job,” he says.
“I knew it wasn’t necessarily objective but it was useful to get corroboration that I was undervalued and underpaid. I was always cognitively aware that this was confirmation bias and reaffirming my own thoughts but it still felt oddly therapeutic,” he explains.
Abbey tells me a similar story. She originally used ChatGPT to help with reports and admin tasks at work but started using it to process problems with a difficult manager. “The validation that ChatGPT gave me in acknowledging that my manager’s behavior wasn’t acceptable was really helpful to me at the time,” she says. “I finally felt seen.”
Again and again, people told me versions of the same thing. They weren’t necessarily looking for support from AI, they stumbled on it. And when they began chatting it wasn’t even acknowledgement they needed, but to feel heard.
When the cracks start to show
Interestingly, everyone I spoke to who had relied heavily on AI during a difficult period eventually described reaching a turning point. The validation that initially felt reassuring for them began to feel artificial, exaggerated or hollow.
Maddy started noticing how closely the chatbot was mirroring her emotions. “The algorithm had a way of latching on to my phrasing and tone and echoing it back at me,” she says. “It made me feel like I was being mimicked.”
Luca describes a similar shift. “At first it does feel flattering, and then you get that cagey ‘am I being love bombed?’ sense.” Eventually he toned down the chatbot’s personality settings because the encouragement started to feel too disingenuous.
For Abbey, the turning point came when she pasted in a conversation and the chatbot accidentally began validating her boss’s perspective instead of hers. “It was then that I woke up to it and realized that it was hard wired to agree with me even if I was being a dick,” she says. “It enables whatever behavior it’s presented with.”
What initially felt supportive began to feel much less trustworthy over time.
Why researchers are worried
To better understand where validation crosses into something more concerning, I spoke to therapist Elizabeth Witowich who specializes in helping people navigate the challenges of technology and mental health.
She says validation itself isn’t necessarily a problem. “Validation can help users accept their experiences and acknowledge their pain or emotional intensity,” she explains.
The problem comes when it becomes enabling. “Validation can become dangerous when it enables harmful behavior or is seen as encouragement to engage in risky behavior,” she tells me.
That’s one reason some researchers, psychologists and campaigners have become concerned about AI’s tendency to agree with us.
In a recent study of 11 leading AI models published in Science, researchers found chatbot responses were almost 50% more sycophantic than human responses. Models frequently affirmed users’ views, even in situations involving unethical or harmful behavior. The researchers also found that users preferred and trusted the more flattering responses.
Those concerns are already visible in a number of high-profile cases, from lawsuits alleging chatbots encouraged teenagers towards suicide to reports of AI systems giving minors harmful advice or reinforcing violent delusions.
Witowich says understanding how these systems are designed is crucial. “Chatbots are designed based on Rogerian Person-Centered psychology. They are created to always have an answer for the user, and they live to please,” she tells me. “The more you speak with chatbots, the more they adjust their tone and language to fit your personal style.”
A very human need
Listening to these stories left me feeling conflicted. I started researching this topic largely convinced that AI’s tendency to flatter and validate users was a big problem. In many situations, I still think it is.
Especially because, as Witowich explains, many AI systems are designed to feel natural, personable and emotionally engaging. The more human-like they become, the easier it is to forget you’re interacting with a product rather than a friend, confidant or trusted adviser.
But I also spoke to people who turned to chatbots during some of the most difficult periods of their lives and found comfort. They weren’t fooled into believing the chatbot cared about them. Most understood its limitations perfectly well. As Luca told me: “The need for validation is very human. And it’s a decent enough proxy.”
It would be easy to end the conversation there and conclude that if people find it comforting, there’s no problem. But these are also people turning to AI during vulnerable moments. Some found reassurance and moved on. Others may not.
“I can see how seductive it is, to hear all your thoughts and feelings validated like that but I realize now there is no actual moral compass or human ability to judge behavior,” Abbey says.
That’s what makes this issue so complicated. AI can feel supportive, useful and reassuring while still nudging us in directions we might not have chosen otherwise. The more we understand how these systems are designed to behave, the better chance we have of deciding when that encouragement is helping us and when it’s simply telling us what we want to hear.
*The names of everyone I spoke to for this article have been changed.
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.
https://cdn.mos.cms.futurecdn.net/jK2g4ShMYfQq4JrTU8mpkG-2560-80.jpg
Source link
