Alibaba Tongyi Lab officially released and open-sourced the film-level multi-scenario voice acting multimodal large model Fun-CineForge on March 16. The model aims to solve core pain points in AI voice acting, such as mismatched lip movements, lack of emotional expression, and inconsistent voice characteristics for multiple characters, and also provides a high-quality dataset construction method.

image.png

In terms of technical architecture, Fun-CineForge introduces the concept of "time modality" for the first time. Unlike traditional models that only focus on text or visual information, this model ensures that the voice is synthesized within the correct time interval through precise timestamp control. Even in complex film scenes where characters are blocked, the camera switches frequently, or faces are blurred, the model can still achieve a very high level of audio-visual synchronization and instruction compliance.

The accompanying open-sourced CineDub dataset construction process is another highlight. Tongyi Lab used large model chain-of-thought technology to automatically convert original film materials into structured data, greatly reducing the cost of manual annotation. Data shows that this process reduces the word error rate to around 1%, and the speaker separation error rate is only 1.20%, providing a highly competitive training foundation for large models.

image.png

Currently, Fun-CineForge has been launched simultaneously on GitHub, HuggingFace, and the ModelScope community, supporting inference for video clips of up to 30 seconds. It not only performs excellently in solo monologue scenarios but also pioneers professional-level support for duet and multi-person dialogue scenarios. This breakthrough marks that AI voice technology is moving from basic customer service and assistant scenarios to high-standard animation and film post-production fields.

  • GitHub: https://github.com/FunAudioLLM/FunCineForge

  • HuggingFace: https://huggingface.co/FunAudioLLM/Fun-CineForge

  • ModelScope: https://www.modelscope.cn/models/FunAudioLLM/Fun-CineForge/