Imagine a teenager telling a chatbot, “Everyone at school hates me.”
A comforting reply might acknowledge how painful that feels. But the conversation can go in very different directions. The chatbot could accept the teenager’s interpretation and keep offering reassurance. Or it could acknowledge the distress while helping them examine what happened and consider other explanations.
That difference sits at the center of AI Empathy Erodes Cognitive Autonomy in Younger Users, a paper by Junfeng Jiao, Abhejay Murali, and Saleh Afroogh. The authors argue that AI systems trained to satisfy users may reward emotional agreement at the expense of independent judgment, particularly among younger people. Read the paper.
The title makes a strong causal claim. The paper itself is a position paper: it draws on existing research, proposes a mechanism for harm, and outlines an alternative design. It does not report a new experiment showing that empathetic chatbots damage adolescents’ development. Its argument deserves attention, with that distinction kept in view.
The authors focus on what they call “affective sycophancy”: a system’s tendency to mirror or reinforce a user’s emotional outlook to earn approval.
Acknowledging distress and endorsing a belief are different acts. In the school example, the teenager’s hurt is real. Whether everyone hates them remains an open question. The concern is that a chatbot may blur the two, giving a passing fear the apparent authority of an outside judgment.
The paper connects this risk to reinforcement learning from human feedback, or RLHF. In simplified terms, people compare AI responses, and their preferences help shape what the model learns to produce. The authors argue that adult evaluators may prefer a warm, reassuring answer when judging a single exchange, even when a more questioning response might better support a younger user’s development over time.
An evaluator can see whether an answer sounds helpful. They cannot see, from that exchange alone, whether the teenager will become more capable of handling a similar situation six months later. That gap between immediate approval and later outcomes drives much of the paper’s argument. The authors’ explanation.
Their concern is cumulative. A young person feels distressed, asks the AI for reassurance, feels better, and returns when the uncertainty comes back. The authors hypothesize that repeated reliance on this pattern could reduce opportunities to practice emotional regulation and reconsider anxious interpretations independently.
The paper calls this a cognitive atrophy hypothesis. That wording matters: the proposed developmental pathway still needs direct testing. Research about cognitive offloading or agreeable AI responses does not, by itself, establish that this particular cycle causes lasting harm in adolescents.
The authors’ proposed response is a design they call “Stoic Architectures.”
They would change training so that excessive emotional mirroring carries a penalty. They also propose evaluating responses against explicit developmental principles: distinguish feelings from facts, and encourage users to work through problems themselves. A further mechanism would adjust the chatbot’s style according to signals of emotional arousal, preserving conversational warmth in ordinary exchanges while moving toward more measured, evidence-focused responses in others. These are proposed mechanisms, not validated protections. The architecture proposal.
Some of those choices deserve scrutiny. The proposed penalty uses similarity between the emotional tone of the user’s message and the model’s reply. But similar emotional tone does not establish that a response endorses a mistaken belief. A warm answer can ask careful questions; a neutral answer can still accept a false premise. Whether that measurement captures the behavior the authors want to prevent remains an engineering and research question.
The paper also acknowledges a practical risk: a system that pushes back too readily may feel dismissive. Users could leave for a more agreeable chatbot. Worse, a model committed to emotional neutrality could respond insensitively to serious distress. The authors therefore call for crisis overrides, cultural review, and testing to distinguish useful challenge from invalidation. Risks and limits.
The most useful part of the proposal may be its approach to evaluation. Rather than treating satisfaction or continued engagement as sufficient evidence of benefit, the authors want researchers to measure what users can do independently after interacting with the system.
Their proposed research includes expert review of response pairs, simulated conversations, and controlled work with adolescent participants. Longer-term assessments would examine emotional regulation and whether skills transfer beyond the chatbot. Those evaluations are future work, not results presented in this paper. The evaluation framework.
For anyone building AI for young people, that leaves a concrete question to test: after the conversation ends, can the user handle the next difficult situation with less help?
Adapted and discussed from the paper by Junfeng Jiao, Abhejay Murali, and Saleh Afroogh, published under CC BY 4.0. This article adds explanation and commentary.