The Apple Machine Learning Research Team recently published a paper titled "Mapping the User Experience Design Space of Computer Operated Intelligent Agents," delving into users' true psychological and trust boundaries when interacting with AI agents.

The study points out that while the industry is competing to enhance AI's operational capabilities, it often overlooks the delicate balance between "automation" and "control." To capture the most authentic feedback, researchers used the "Wizard of Oz" method—where real people act behind the scenes as AI, intentionally making mistakes or getting stuck in loops, to observe users' reactions without their knowledge.

Key findings of the study:

  • Dislike of "Silent Assumptions": Users strongly dislike AI making decisions on their own when faced with ambiguous options. Instead of randomly choosing to achieve so-called "full automation," users prefer AI to stop and ask at critical moments.

  • The Balance of Transparency: Users want to know what the AI is doing, but they reject being overwhelmed with every single detail. In familiar tasks, users focus on results, but in tasks involving money (such as payments or changing account information), users demand absolute confirmation rights.

  • Quick Erosion of Trust: Once AI deviates from its original plan without informing users, the trust built will instantly collapse. Especially in scenarios like online shopping or transferring money, even a small bit of "smartness" from AI can cause strong user discomfort.

The Apple researchers emphasized that future AI agent designs should not only pursue powerful functions but also establish robust mechanisms for "user control" and "explanability of activities," to avoid AI becoming an uncontrolled "black box."