The development team of the open-source code editor Zed released a widely discussed blog post titled "Why LLMs Can't Truly Build Software." Written by Conrad Irwin, the article sparked intense discussions in the tech community, especially on Hacker News, where many developers shared their opinions focusing on the essence of AI-assisted programming.

Irwin points out that excellent software engineers go through a cyclical process when working, including building a mental model of requirements, writing code, understanding the actual behavior of the code, and identifying discrepancies and making updates. He emphasizes that the ability to maintain a clear mental model is the key differentiator between good engineers and others. However, current LLMs (large language models) have a critical flaw in this aspect.

He argues that while LLMs perform reasonably well in writing and updating code, they cannot maintain a clear mental model. When tests fail, LLMs often become confused, assuming the code they wrote is correct but not knowing how to fix it, eventually choosing to rewrite the code. Many developers agree with this view, believing that LLMs often lose sight of the overall architecture when handling complex projects.

However, the article also sparked differing opinions. Some developers believe that today's LLMs and coding tools are already quite powerful, sometimes even surpassing junior engineers. Some users shared their experiences of using AI to write projects, saying the results were surprising, even completing a 7000-line code project entirely with AI.

Irwin further analyzed three major issues LLMs face: context omission, recent bias, and hallucination. These problems constrain LLMs in maintaining a mental model. In contrast, humans can effectively store and process complete context, allowing for greater flexibility and adaptability when solving problems.

The Zed team's final conclusion is that although AI can assist in software development, humans remain the main force in the process. As the team stated, "At least for now, you're still the driver, and the LLM is just a tool you can use." Many developers agree, believing that AI can handle some tedious coding tasks, but design and planning still rely on human intelligence.