The latest five experiments involving 3,132 participants
The research team specifically selected highly obscure questions that are prone to model hallucinations for testing. The results showed that in the control group without AI assistance, participants chose not to answer 36% to 44% of the questions; however, after obtaining AI suggestions, this non-response rate dropped sharply to 3% to 6%.

More paradoxically, participants' confidence in answering questions increased from 29.6 points (out of 100) to 75.9 points with AI assistance, but their actual accuracy rate dropped dramatically from 27.5% to 9.2%, indicating that even when faced with incorrect suggestions from models like Step3.5Flash, people tend to blindly accept them and develop excessive confidence.
Introducing economic incentive mechanisms such as rewards for correct answers and deductions for incorrect ones can reduce the frequency of people seeking AI help to some extent, but it has not fundamentally reversed this trust misalignment. More concerning for the industry is that when the test simulated the current norm of search engines and writing assistants by automatically displaying AI-generated answers to users, even under economic incentives, the proportion of participants suspending judgment dropped from 39% to 7%.
Researchers attribute this phenomenon to a new trend of blind trust, where humans are easily persuaded by AI's fluent expressions, thus breaking the traditional habit of retaining some independent judgment in the process of seeking help. Combined with previous findings from Microsoft and related academic institutions regarding AI's impact on human critical thinking, over-reliance on AI is leading people to outsource cognitive tasks excessively.
As generative AI becomes seamlessly integrated into various digital tools, how to improve the accuracy of models while designing appropriate human-computer interaction mechanisms to maintain human awareness of their own knowledge boundaries will become an important issue in the next phase of AI application.
Join Now