NVIDIA's research team has made a breakthrough: they have found a way to migrate KV caches between different models. The target model can directly skip the pre-filling phase, with a conversion speed 2.7 to 25 times faster than reprocessing the context.

To understand the significance of this breakthrough, it is necessary to first understand the role of KV cache. When using ChatGPT or Claude, you will notice that the generation of the first word is always significantly slower, and then the subsequent content almost instantly flows out. This is an intentionally designed mechanism — the model needs to process the entire input context before generating the first word, storing the intermediate results as KV cache. Each subsequent word can reuse these caches, so it is much faster.

The problem is that KV cache has always been "one model at a time." If you switch to another model, or upgrade the model version, the previously computed cache becomes useless, and the new model must process the entire context from scratch. For long text scenarios, this means a lot of redundant calculations and wasted time.

NVIDIA's solution makes KV cache transferable. A cache computed by one model can be directly used by another model after conversion, allowing the target model to completely skip the pre-filling phase. The conversion process itself is also much faster than reprocessing the context — the speed improvement ranges from 2.7 times to 25 times, depending on the model and the complexity of the context.

The impact of this on the AI industry is profound. A significant portion of the current cost of using LLM APIs comes from context processing, especially in long conversation and long document scenarios. If KV cache can be migrated between models, it means users don't have to bear the cost of redundant calculations when switching models, and model upgrades don't require discarding existing conversation contexts. This research may change the way AI inference infrastructure is designed, making model switching more seamless and economical.