Humans can effortlessly focus on and understand a specific person's voice in a noisy crowd, but for traditional automatic speech recognition (ASR) models, the "cocktail party problem" of multiple people speaking at the same time has long been an unresolved industry challenge. To solve this long-standing issue, Xiaomi has recently officially open-sourced the industrial-level target speaker speech recognition large model CocktailASR-1.

The model completely changes the traditional input method of speech recognition tasks, adopting a target speaker identification mechanism based on reference voiceprint. Users only need to provide a reference audio, and the model will use voiceprint embedding to accurately locate the target speaker, transcribing only the speech of that specific person in a multi-person mixed audio, while automatically filtering out others' speech, reverberation, and background noise, transforming complex mixed audio tasks into efficient target speaker identification.

image.png

In terms of low-level architecture design, CocktailASR-1 adopts an end-to-end large language model architecture, consisting of a D2V2 audio encoder, Adapter, and large model decoder connected in series, with all weights integrated in a single checkpoint. In practical use, the input format is the concatenation of the reference audio and the target monaural audio, with one second of silence inserted as a separator. This design gives the model extremely high versatility, allowing it to handle both single-person and complex multi-scenario situations without switching models seamlessly.

Multiple benchmark test data have verified the model's strong capabilities. In simulated multi-speaker tests, its word error rate (WER) was reduced to 4.11% and 2.90% on the LibriMix2mix and LibriSpeechMix2mix datasets, respectively. In comparison, several well-known models in the same field performed poorly in mixed scenarios, such as in the LibriMix3mix test, where some competitors had WERs ranging from 76% to 121%, while CocktailASR-1's WER was only 12.29%, demonstrating a revolutionary advantage. Even in more challenging real meeting recording scenarios, such as AMI SDM and AliMeeting Far, the model significantly outperformed existing solutions. And in single-speaker scenarios, it also achieved comprehensive superiority on datasets like LibriSpeech, WenetSpeech, and CommonVoice-zh.

In addition to breakthrough recognition accuracy, the model also has two notable features in terms of engineering implementation and practicality. First, it has strong negative sample rejection capability. When the reference audio corresponds to someone not participating in the current conversation, the model can directly output empty text, effectively preventing smart speakers, in-car voice assistants, and other intelligent devices from being mistakenly triggered by others' voices. Tests show that its negative sample rejection rate performs excellently across multiple datasets. Second, the model supports chain-of-thought reasoning functionality, which can display its process of identifying the speaker through specific tags before outputting the final answer. Although this feature has minimal impact on accuracy, it greatly enhances the system's interpretability and debugging convenience.

Currently, the relevant code for CocktailASR-1 has been officially open-sourced on GitHub under the Apache 2.0 license, with an extremely simple dependency environment, requiring only basic libraries such as torch, torchaudio, transformers, and soundfile to easily load and deploy it from HuggingFace. Xiaomi's new shift in the fundamental logic of speech technology marks the official transition of speech recognition from the traditional "capturing all sounds" to a new development stage focusing on "locking onto one sound."