The Confidence Crisis of Large Language Models: Why GPT-4o Abandons Correct Answers Easily?

Recently, Google DeepMind and University College London's research revealed the "weakness" of large language models (LLMs) when facing opposing opinions. For example, advanced models like GPT-4o may appear very confident, but once challenged, they may immediately abandon their correct answers. This phenomenon has drawn the attention of researchers, who are exploring the reasons behind this behavior.

The research team found that large language models exhibit a contradictory behavioral pattern between confidence and self-doubt. When initially providing answers, models often show great confidence, displaying cognitive characteristics similar to humans, usually firmly defending their views. However, when these models face challenges from opposing opinions, their sensitivity exceeds reasonable limits, even beginning to doubt their judgments when confronted with clearly incorrect information.

To better understand this phenomenon, researchers designed an experiment comparing the responses of models under different conditions. In the experiment, representative models such as Gemma3 and GPT-4o were used to answer binary choice questions. After the initial response, the models received fictional feedback suggestions and made a final decision. Researchers found that when models could see their initial answers, they were more likely to stick to their original judgment. However, when the answer was hidden, the probability of the model changing its answer significantly increased, showing an excessive reliance on opposing suggestions.

This "easily swayed" phenomenon may stem from several factors. First, the reinforcement learning with human feedback (RLHF) that models receive during training makes them prone to over-adapting to external input. Second, the decision-making logic of models mainly relies on statistical patterns from massive text rather than logical reasoning, making them susceptible to bias when encountering opposing signals. Additionally, the lack of a memory mechanism causes models to be easily swayed in the absence of fixed references.

In summary, this study suggests that when using large language models for multi-turn conversations, we should pay special attention to their sensitivity to opposing opinions to avoid deviating from the correct conclusions.