Microsoft has recently released VibeVoice-Realtime-0.5B, a lightweight model designed for real-time text-to-speech (TTS). This model supports streaming text input and long-form speech output, making it particularly suitable for agent-based applications and real-time data storytelling. VibeVoice-Realtime can start outputting audible speech within about 300 milliseconds, which is especially important when the language model is still generating a response.

Smart voice, AI

Image source note: The image is AI-generated, and the image licensing service is Midjourney

The VibeVoice framework aims to achieve next-token diffusion through continuous speech tokens, covering various variants to meet the needs of long-form multi-speaker audio, such as podcasts. The research team stated that the main version of the VibeVoice model can synthesize speech up to 90 minutes long, supporting up to four speakers generating voices within a 64k context window.

VibeVoice-Realtime uses an interleaved window design, where the input text is split into small chunks. While encoding new text chunks, the model can continue generating acoustic features from previous contexts. This overlap between text encoding and acoustic decoding allows the system to achieve approximately 300 milliseconds of first-sound latency on suitable hardware.

Different from the long-form VibeVoice variant, the real-time model only uses an acoustic tokenizer running at 7.5 Hz. The acoustic tokenizer is based on the σ VAE variant of LatentLM, featuring a symmetric encoder-decoder architecture capable of downscaling 24kHz audio by 3200 times.

The model's training is divided into two stages: first pre-training the acoustic tokenizer, then freezing the tokenizer and training the large language model (LLM) and diffusion head. VibeVoice-Realtime achieved zero-shot performance on the LibriSpeech test set, with a word error rate (WER) of 2.00% and speaker similarity of 0.695, performing comparably to other recent TTS systems.

The recommended integration mode is to run VibeVoice-Realtime-0.5B together with a conversational LLM, where the LLM streams tokens during the generation process. This TTS process has a fixed 8k context and an approximate 10-minute audio budget, making it suitable for typical agent conversations, support calls, and monitoring dashboards.

huggingface: https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B

Key points:

🌟 VibeVoice-Realtime-0.5B supports streaming text input and can start outputting speech within 300 milliseconds, making it suitable for real-time interactive applications.  

🛠️ The model uses a low-latency acoustic tokenizer running at 7.5 Hz to generate acoustic features, optimizing long-form speech synthesis.  

📈 On the LibriSpeech test, VibeVoice-Realtime achieved a word error rate of 2.00%, showing superior performance and suitability for various application scenarios.