The foundation of large model training is being quietly shaken by a Chinese company. In the speech "How We Scaled Kimi K2.5" at GTC2026, Moonshot CEO Yang Zhilin did not just talk about K2.5 as a product, but laid out a clear understanding the company has had over the past year: when open-source models approach closed-source frontiers, there's another path besides continuing to increase parameters and computing power — re-implementing each of the three fundamental components used in the Transformer era for nearly a decade. Optimizer Adam (proposed in 2014), attention mechanism (2017), and residual connection (2015). Moonshot has provided alternative solutions for each of these foundational components, and all are open-sourced.

QQ20260720-095005.png

Yang Zhilin broke down the entire technical roadmap into three directions: making each token more valuable, making longer context truly useful, and enabling multiple agents to collaborate simultaneously. In addition, he also introduced two important developments: how visual training can enhance text capabilities, and Moonshot’s newly announced next-generation architecture, Attention Residue.

First, the most pressing wall: high-quality data is running out. There is already limited valuable text on the internet, and as models grow larger, the amount of data needed increases, and everyone will eventually hit a data wall. Since it's hard to double the data, Moonshot's approach is to let the model learn more from the same data. The solution is MuonClip, which replaces the Adam optimizer that has been used for over a decade. Based on the Muon optimizer, it tries to keep different directions of information independent during parameter updates, reducing repetition and waste, thus improving the efficiency of data usage. Experimental results show that with the same amount of data, MuonClip's training effect is close to doubling the data volume, which lowers the training cost and raises the model's capability ceiling. Yang Zhilin gave an analogy: if you have 50 trillion high-quality tokens, doubling the efficiency equals having another 50 trillion tokens. In today's context where high-quality data is becoming increasingly scarce, such efficiency improvements are more critical than simply adding computing power.

However, Muon has a problem: when the model scales to trillions of parameters, the values in the attention layer can become unstable, with the maximum logit suddenly spiking above 1000, while the normal range is usually between 50 and 100. If the values continue to expand, the training curve may diverge, and the entire training process could collapse. To address this, Moonshot designed QK-Clip, which checks the maximum logit of each attention head in real time during forward computation. If it exceeds the safe range, it synchronously scales down the Q and K projections to bring the values back under control. It doesn't change the convergence effect, only maintains numerical stability. With QK-Clip, Muon was successfully scaled to trillions of parameters, trained on over 15 trillion tokens, and no loss spikes occurred during the entire process.

The second direction is making long context truly useful. Yang Zhilin cited experiments by Jared Kaplan and others from the Scaling Law paper: LSTM cannot improve beyond a certain length, but Transformers perform better with longer context and rarely show saturation. This characteristic becomes particularly important in the Agent era — complex tasks may last days or even weeks, and the model must remember what it has done, the results obtained, and which directions have failed. If long context fails to continuously provide effective information, the task chain is likely to break. The problem is that the computational cost of standard full attention grows quadratically with the context length. Increasing the length by ten times can increase the cost by a hundredfold, making training and inference costs extremely high at a million tokens. Moonshot's solution is Kimi Linear, which features a linear attention mechanism called KDA (Kimi Delta Attention). It enables the model to learn which information should be retained long-term and which should be quickly forgotten. Traditional linear attention uses a single global decay coefficient, like a universal forget button for all content; KDA splits this decay coefficient into multiple parts, allowing different information channels to have different forgetting speeds. Channels with slow decay retain long-distance information, while those with fast decay quickly make space for new content. In practice, Kimi Linear mixes linear attention layers and full attention layers in a 3:1 ratio. Yang Zhilin claims this is the first architecture that comprehensively outperforms full attention in short context, long input, and long output tasks. When context extends to millions of tokens or more, the efficiency advantage becomes even more apparent.

The third direction is evolving from one agent to a team of agents, which Moonshot calls Agent Swarm. Its structure resembles a company: a main agent acts as the CEO, responsible for understanding the goal, breaking down tasks, and assigning them to multiple sub-agents. Sub-agents can take roles like researchers, programmers, data analysts, and fact-checkers. Once completed, the main agent summarizes the results. The greatest value is transforming serial tasks into parallel ones. Previously, a single agent would sequentially search, read, analyze, code, and verify. Now, dozens or even hundreds of agents can work simultaneously. Training such collaboration is not easy. Moonshot designed three types of rewards: instance-based rewards encourage the main agent to create more parallel sub-tasks and avoid reverting to serial processing; completion rewards require sub-tasks to be genuinely completed, preventing the creation of empty tasks for reward inflation; and final result rewards determine whether the task was actually solved. The weights of these rewards dynamically change during training, focusing on task decomposition and parallelization in the early stages and shifting to final results in later stages. From demonstrations, Agent Swarm can simultaneously download and read hundreds of information sources for research, parallelly write different chapters of a hundred-page literature review, and simultaneously analyze ten datasets. When Kimi K2.5 was released, it supported up to 100 agents working in parallel, with tasks executing up to 1,500 steps. The K2.6 version released in April increased the parallel limit to 300 agents.

An unexpected gain emerged in the fusion of vision and text. The key difference between K2.5 and its predecessor K2 is the use of early fusion training: many open-source multimodal models previously used late fusion, first training a language model with around 20 trillion text tokens, then adding vision capabilities with about 2 trillion multimodal tokens. K2.5, however, mixed visual and text data from the very beginning, training on about 15 trillion mixed tokens on top of the K2 text base. This led to two exciting results for Yang Zhilin. First, training only on visual tasks can also improve text reasoning: the research team only let the model count, recognize images, and answer visual questions, without adding math or programming training, yet the text reasoning ability improved. In other words, the model improved its thinking ability while learning to see. Second, in reverse, if the text base is strong enough, the model might not even need specific visual SFT data — K2.5 adopted a zero-vision SFT approach, using only pure text for all supervised fine-tuning data, and then gained visual capabilities through joint reinforcement learning of text and vision. Ultimately, it achieved near state-of-the-art performance on visual tasks. Yang Zhilin believes this bidirectional transfer comes from early fusion: when text and vision enter the same representation space, the ability learned in one modality has a chance to transfer to another. This is the foundation of K2.5's ability to write code based on images.

At the end of the speech, Yang Zhilin introduced Moonshot's newly released research, Attention Residue, published on March 15th, just two days before the speech. Residual connections, proposed by He Kaiming et al. in 2015, are a fundamental technique in deep networks and now a standard component of Transformers. They allow each layer to process the upper layer's result while maintaining a direct channel for information transmission, enabling stable training of deep models. Yang Zhilin referenced Ilya Sutskever's view, interpreting residual connections as LSTM rotated 90 degrees — while LSTM transmits information along the time dimension, residual connections transmit it along the depth dimension. Since Transformers have replaced LSTM's cyclic function on the time dimension with attention, can the depth dimension also be similarly replaced? Attention Residue is the outcome of this idea: standard residuals mainly use the output of the previous layer, while it allows the current layer to access all previous layer outputs, then uses attention to decide which historical information to retain and which to ignore, letting the model actively select information from the entire computational history instead of passively receiving the upper layer's result. To control costs, the block version Block Attention Residual is used, where every 16 layers form a block — within blocks, standard residuals are still used, and attention residuals are applied between blocks. Experiments show that about 8 blocks can capture most of the benefits. It brings a 24% improvement in Token efficiency. According to Yang Zhilin's algorithm, a 24% efficiency gain on 50 trillion Tokens equals an additional 12 trillion Tokens. This improvement is especially evident in tests such as GPQA, MATH, and HumanEval, which focus on reasoning, mathematics, and programming.