Reduce the First Token Latency by 3.25 Times: Xiaohongshu Collaborates with Peking University and Shanghai Jiao Tong University to Propose HYPIC, Equipping Hybrid Attention Large Models with Location-Independent Caching

The main battlefield of large model services is shifting from single-turn chat to retrieval-augmented question answering, multi-document summarization, and long-range agents. These new workloads share a common characteristic: a single request's prompt is often composed of dozens to hundreds of semantically independent segments, with retrieved documents, skill descriptions, memory files, and previous conversation rounds all being assembled into ultra-long contexts of tens of thousands or even hundreds of thousands of tokens. At this length, the prefill phase dominates the computational cost per request, becoming the most visible cost for service providers; more problematically, when a cache miss occurs, the tail-first token delay (TTFT) can surge to several seconds, directly ruining the user experience.

The industry has developed two cost-reduction approaches. One is position-agnostic caching (PIC), which relaxes strict prefix constraints, allowing each semantically independent segment to be cached once and then spliced to any prefix, perfectly matching the prompt composition method of RAG and agents. Its essence can be summarized as two primitives: splice, which directly concatenates along the token axis, and correction, which recalculates a few tokens to restore cross-segment context. The other approach is the hybrid attention model, which replaces most full attention layers with linear attention, reducing the quadratic complexity of attention to linear, and compressing unbounded history into a fixed-size cyclic state. Recent production models such as MiniMax-M1, Ring-2.5, Qwen3.5, and Kimi-Linear generally linearize over 75% of their layers, retaining only a few full attention layers, forming the mainstream hybrid attention design. For example, Qwen3.5-35B-A3B has 30 linear layers out of 40 total layers.

The contradiction lies precisely here: existing PIC operates on per-token key-value (KV) caches, while linear attention layers only expose a per-request cyclic state, without any per-token hooks available for splice or correction. As a result, most layers in hybrid models fall outside the capabilities of existing PIC systems. Before this, no system could provide position-agnostic caching for hybrid attention large models.

Developed by the Xiaohongshu large model inference team in collaboration with Peking University and Shanghai Jiao Tong University, HYPIC is the first service system to implement PIC on hybrid attention large models. Testing on four production-level hybrid attention models and five workloads shows that the first token delay is reduced by an average of 3.25 times, and under the same SLO, sustainable QPS increases by 1.66 times, with task quality differing by only 1.71 points from full recomputation. The core idea of this system is to handle linear and full attention layers separately within the hybrid attention model, and then accelerate cold requests through a layer-level parallelism, consisting of three interlocking mechanisms.

For linear layers, HYPIC caches "transition operators," enabling constant-time state combination. The most straightforward PIC straw man solution is to directly add the final states of two segments with zero initial values—this holds true in naive linear attention, but fails in advanced linear attention with decay, gating, or delta erasure (e.g., RetNet, Mamba2, GLA, DeltaNet, GDN, KDA). The critical quantity missed is the accumulated transition operator T_C of a segment, i.e., the product of all token transition matrices within the segment. The actual final state should be S(C1·C2) = T(C2) · S(C1) + S(C2), while simple addition omits the T(C2) term, causing structural error. In tests, the slow decay head in RetNet showed an error of 22% of the state norm at 256 tokens. HYPIC’s key insight is that T_C and the zero-initial-state final state S(C|0) are determined solely by the internal tokens of a segment, independent of the prefix. Therefore, during the first prefill of a segment, HYPIC caches the pair (T_C, S(C|0)), and upon reuse, applies left multiplication of T_C followed by addition to nearly exactly reconstruct the final state under any prefix, naturally covering the entire linear attention family. In tests on Qwen3.5-35B-A3B, the combined state differed from full recomputation by only 6×10⁻⁵, within FP16 noise. For variants with causal convolution or RoPE, HYPIC uses two patches—convolution state warm-up and state re-rotation—to achieve strict alignment.

For full attention layers, HYPIC uses "seam windows" to fix cross-segment attention. Although a few full attention layers still require PIC in hybrid models, previous selective recomputation cannot be directly migrated because the downstream linear layers retain only the final state, making it impossible for non-terminal tokens to pass upward through the full attention layer. Two alternatives—storing the cyclic state per token or recomputing from scratch—are not feasible in terms of storage or computation. The team observed that the bias after KV concatenation is highly concentrated at the beginning of each reused segment, with the rest almost unaffected, consistent with the attention sink phenomenon in full attention models, which also holds in hybrid attention models. Based on this, HYPIC constructs a seam window of w tokens at the beginning of each internal segment, excluding these tokens from caching. Upon reuse, they are recomputed under the complete prefix to repair cross-segment attention, with w=8 by default; since the actual segment length is usually greater than 512 tokens, the seam window constitutes a very small portion, making the recomputation cost manageable. To support the recomputation of the initial KV, the linear attention layers compute the T and S of the seam window during reuse, advancing the running state while forwarding the per-layer outputs of each seam token to the upper full attention layer.

The third piece of the puzzle is segment-level parallelism, which transforms "long cold requests" into accelerated workloads. Cache misses are inevitable due to continuous corpus updates and low-frequency document removals, and long cold requests are the main cause of tail latency. Existing systems still treat the entire prompt as a single entity, serially prefilling all cold segments on a single instance, with TTFT growing as O(n·|C|); although tensor parallelism and other intra-instance parallelisms can speed up individual forward passes, their scalability is limited, and a 100k-token request on TP-8 still takes 17.7 seconds. However, PIC has already made each segment self-contained—its prefill results depend only on its internal tokens. HYPIC accordingly proposes inter-instance segment-level parallelism: the Router detects whether each segment hits the cache, then distributes the cold segments in parallel to multiple scatter workers, and a combine worker assembles the results of each segment into a complete running state, reducing the cold prefill time from O(n·|C|) to O(⌈n/m⌉·|C|+c). To fully utilize this path, HYPIC uses a greedy LPT (Longest Processing Time) strategy to balance the workload across workers, and overlaps the computation and transmission of each segment to avoid the combine worker being held back by synchronous transmission bursts; segment-level parallelism and intra-instance parallelism (TP/DP/SP) operate on orthogonal axes and can be seamlessly combined.

This system has been implemented on SGLang, with approximately 14,000 lines of Python and Triton code. It was evaluated on eight H20 nodes using four production-level hybrid attention models (Ring-mini/flash-linear-2.0, Qwen3.5-35B-A3B/122B-A10B), four public datasets (HotpotQA, TriviaQA, MultiNews, GovReport), and one production RAG trace. After full preheating, compared to Prefix Cache, HYPIC reduces p50 TTFT by an average of 3.25 times (from 2.77× to 4.05×), with almost no loss in quality—16 "model × dataset" units averaged only 1.71 points behind full recomputation, and even exceeded it by 0.47 points on Qwen3.5-35B. As a comparison, a naive addition without caching transition operators lost 66.9% of the score, confirming the necessity of T_C. On the production RAG trace, under the same 1-second TTFT SLO, HYPIC increased sustainable QPS by 1.66 times (from 1.49× to 1.85×), and peak throughput per card increased by about 1.3 to 1.5 times. On the pure cold miss path, a 32k-token request's TTFT dropped from 2.83 seconds on a single worker to 0.49 seconds on eight workers, achieving a 5.7× speedup, with nearly all gains coming from the scalable scatter stage. Against uneven sharding, LPT reduced TTFT from 1.26 seconds to 0.84 seconds (3.6× vs. 2.4×). The overhead is also manageable: the one-time cost of constructing (T_C, S(C|0)) is only 5.2% to 6.7% of the main forward pass in the heaviest dense transfer model, and this cost can be amortized by reusing it multiple times.

HYPIC brings position-agnostic caching to hybrid attention large models for the first time: it enables constant-time state combination for linear layers using cached segment-wise accumulated transition operators, fixes cross-segment alignment for full attention layers with small seam windows, and turns long cold requests into accelerated workloads using segment-level parallelism. This allows long-context services like RAG and agents to enjoy the benefits of segment-level cache reuse while embracing efficient hybrid architectures. The team stated that future work will focus on exploring distributed PIC management and scheduling, and multi-level cache hierarchy management, further advancing large model inference toward faster, more cost-effective, and more scalable solutions.