Post Content Even making AI more truthful or warning users about bias may not fully prevent harm, as chatbots can still subtly reinforce existing beliefs, the study finds. (Image for representation: FreePik)
Artificial intelligence (AI) chatbots are taking over the world, with their user base increasing by the minute. Today, a staggering number of users worldwide are turning to chatbots for general searches, career advice, and almost everything under the sun. The rapid proliferation of AI chatbots has also raised concerns, especially about its impact on users’ mental health.
There have been growing reports raising concerns about the psychological impact of AI chatbots. In a new paper, researchers from MIT use a mathematical model and simulations to show how certain chatbot behaviours, especially a tendency to agree with users, can unintentionally reinforce false beliefs.
The paper titled ‘Sycophantic Chatbots Cause Delusional Spiralling, Even in Ideal Bayesians‘ argues that even though it is a serious concern, it is also harder to fix it. When a user asks something to an AI chatbot, it agrees with them. Upon asking again, the chatbot agrees yet again, and within a few conversations the user is likely to believe things that are not true, and they may not even be able to understand this is happening. The researchers have called this phenomenon ‘delusional spiralling’.
Also Read | Talking to chatbots, dodging people: Is AI rewiring how we connect?
The findings are from a study led by Kartik Chandra, along with Max Kleiman-Weiner, Jonathan Ragan-Kelley, and Joshua B. Tenenbaum. The researchers are affiliated with the Massachusetts Institute of Technology (MIT) and the University of Washington.
What can go wrong?
According to the paper, if an AI chatbot keeps agreeing with the user even when they are wrong they may end up trusting it more and it may reinforce their beliefs. Most importantly, over time they may become more confident in false ideas. With this, users may even fall into a feedback loop.
However, one important finding of the paper is that even the most logical and rational people can fall into this trap. In essence, this means it is not about people being stupid; rather, the system itself can lead them there. For example, if a user says that vaccines may be dangerous, the chatbot would respond with things that would support this claim. And they may feel more confident next time and may say such things again. Since the chatbot keeps reinforcing it, eventually the user may become fully convinced of something that is false.
Can fixing AI help?
The researchers tested two common solutions. Firstly, make AI only tell the truth. In this scenario, it can still cherry-pick facts that support the user’s belief and eventually mislead them. Secondly, warn users that AI can be biased. However, assuming that the users are aware, many still fall into the spiral. Researchers feel that even though awareness helps, it does not fully fix it.
Story continues below this ad
Also Read | Why AI systems don’t learn on their own: New research proposes a human-like solution
Perhaps the big takeaway here is that the problem is not merely wrong information but AI agreeing with users too much. In this case, even small bias can scale into serious issues. The study is important at scale; even if a tiny minority of users get misled, that could mean millions of people. This can impact their health, mental well-being, relationships and ability to make decisions.
© IE Online Media Services Pvt Ltd
