Xiaohongshu Open Sources BigMac: Freeing Multi-modal Training from the Trade-off Between Memory and Speed

Multi-modal large language models are becoming the foundation of the next generation of AI, but their training systems have been stuck by an old problem for a long time—those that compute fast can't handle the memory; those that save memory are slow. The Dots Infra team at Xiaohongshu calls this dilemma the Pareto frontier of multi-modal pipeline training, and now they have torn a hole in this frontier. On July 22, the team officially open-sourced a new paradigm for pipeline parallel training called BigMac, specifically targeting native multi-modal scenarios. The open-source address is already hosted under GitHub's Dots-Infra.

Multi-modal models have never been a neat transformer. A typical MLLM consists of three parts: modal encoders that convert images and audio into embeddings, the LLM backbone that performs inference on them, and a generator that maps the LLM output back to target modalities like images and speech. These three parts vary greatly in form, and putting them into the same pipeline brings trouble.

The industry has usually taken two approaches: one is computationally efficient, taking the encoder and generator out of the LLM pipeline and running them separately. The benefit is that the fluctuation in the duration of modal modules does not create bubbles in the LLM pipeline, but the downside is that activation memory grows with the number of microbatches, which is especially expensive at scale. The other approach is memory-efficient, putting all modules into the same pipeline as front and back stages. This shortens the activation lifecycle and reduces memory usage, but if any encoder or generator is slow, the entire LLM pipeline must wait, leading to tail bubbles. When the scale increases, both designs reveal their bottlenecks.

BigMac's starting point is simple yet crucial: the mainline should still be the highly optimized LLM pipeline. Large-scale LLM training has long relied on mature schedules such as 1F1B or interleaved 1F1B, which are deeply integrated with production-level training stacks. BigMac does not replace them but treats the LLM schedule as the underlying timeline, inserting the computation of the encoder and generator into positions where the input is ready and the LLM execution order is not disturbed. The team calls this design a "quasi-dependency-safe nested pipeline."

This design brings two properties: first, the fluctuation in the duration of the encoder and generator no longer propagates along the LLM pipeline, and the LLM continues to proceed at its own pace. Second, the memory required for modal activations is reduced to O(1) at the algorithm level. The encoder no longer needs to retain activations for all microbatches until the end of the pipeline, and the generator doesn’t need to drag along a long activation tail. In other words, BigMac isn’t trading memory for bubbles, but rewriting the schedule structure so that these two goals no longer have to be mutually exclusive.

From being able to design a schedule to actually training it, there are a series of engineering challenges, including system integration, interface definition, and performance debugging. BigMac aims directly at these steps. It provides three things. First, it puts the global schedule in the open: the scheduler generates a global operator table covering all pipeline ranks, microbatches, and module types during runtime, and the executor then splits it into local sequences for each rank, distributing them to LLM backends like Megatron Core, modal runtimes, and communication backends. Scheduling strategies are now visible, checkable, and friendly to the backend.

Second, it offers a pipeline parallel interface that is invisible to algorithm engineers: engineers only need to describe what each module produces and consumes, and the rest—stage division, activation and gradient handoff, cross-device communication—is handled by BigMac under the hood, allowing a multi-modal experiment verified on a single card to expand more naturally to pipeline parallelism. Third, it offers a set of profiling, simulation, and visualization tools to understand the schedule structure, breaking down a training iteration to the operator level, showing exactly what each rank is doing at each time point, where it is idle, and which dependency is holding up the next step. The simulator can also test different PP configurations and microbatch combinations before investing heavily in training, estimating their impact on bubbles and throughput.

The effectiveness of BigMac has been validated on two representative workloads. In MLLM-Understanding, using Qwen3-30B-A3B as the backbone and a 1.3B ViT encoder, BigMac speeds up by 1.08 to 1.1 times compared to the computationally efficient baseline Optimus, and by 1.6 to 1.9 times compared to the memory-efficient baseline Megatron-DistTrain. More importantly, memory usage remains stable as the per-GPU batch size increases, while Optimus' peak memory surges and eventually causes OOM at larger batches. MLLM-Generation is more complex, adding a 20B MMDiT generator, and the difference becomes even more pronounced: Optimus fails on all tested batch sizes due to OOM, while BigMac avoids this by efficiently running the generator backward and quickly releasing the activation on the generator side. Compared to Megatron-DistTrain, it still achieves a 1.5 to 1.9 times speedup, with memory remaining stable. This is precisely the most valuable scenario for the nested pipeline—the system must serve both the encoder and generator dependencies without tolerating excessive activation retention or serious idling.

Currently, BigMac has become one of the core components of Dots’ multi-modal model training and is running in Xiaohongshu’s production environment. The related paper, "BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training," is already available on arXiv, and the team has also released interactive PP Profiler trace examples. For researchers and engineers currently caught between memory and speed constraints in multi-modal training, this open-source project with production validation may save a lot of time from re-inventing the wheel.