Alibaba Group has once again made strides in digital human technology by officially open-sourcing MNN TaoAvatar, a 3D digital human application built on the MNN framework. This innovative technology brings highly realistic 3D virtual image generation and real-time interaction capabilities to mobile devices, opening up new possibilities for fields such as live streaming, virtual social networking, and AR applications. Below, AIbase will provide you with a detailed analysis of this remarkable technological breakthrough.

MNN TaoAvatar: The Magic Wand for 3D Digital Humans on Mobile Devices

MNN TaoAvatar is Alibaba's 3D digital human technology developed based on its open-source lightweight deep learning inference framework MNN. Unlike traditional 2D Live2D technology, MNN TaoAvatar supports real-time generation and driving of true 3D virtual characters, running at up to 90 FPS on mobile devices like smartphones, providing a smooth interactive experience.

This technology combines 3D Gaussian Splatting techniques to generate photo-realistic 3D full-body virtual images from multi-view image sequences. Whether it’s facial expressions, gestures, or body postures, MNN TaoAvatar can achieve millimeter-level fine control, ensuring that the virtual character’s lip movements, expressions, and actions are naturally synchronized, offering users an incredibly lifelike visual experience.

image.png

Technical Highlights: Lightweight and Efficient, Multi-modal Drive

The success of MNN TaoAvatar is inseparable from the strong performance support of the MNN framework. As a reasoning engine open-sourced by Alibaba since 2019, MNN is widely praised in the industry for its lightweight, high-performance, and cross-platform compatibility. On this basis, MNN TaoAvatar further optimizes, possessing the following core advantages:

Real-time facial capture: Through deep learning algorithms, MNN TaoAvatar can accurately capture users' emotions of joy, anger, sorrow, and pleasure and synchronize them to 3D virtual characters with low latency, suitable for real-time interactive scenarios such as live streaming and virtual meetings.

Lightweight deployment: Thanks to MNN's model quantization and memory optimization technologies, MNN TaoAvatar can run smoothly on ordinary phones without requiring high-end hardware, significantly lowering the threshold for use.

Multi-modal support: In addition to facial expression capture, MNN TaoAvatar also supports multiple input methods such as voice, text, and image generation, providing developers with rich creative space.

Open-source ecosystem: As part of Alibaba's open-source strategy, MNN TaoAvatar provides complete APIs and tools, making it easy for developers to integrate it into Android and iOS applications, helping with rapid development and deployment.

In addition, MNN TaoAvatar optimizes non-rigid deformation processing through knowledge distillation techniques and learnable Gaussian mixture shape modeling, ensuring that the virtual image maintains high fidelity even under complex poses. This technical innovation allows it to achieve high-quality rendering on resource-constrained mobile devices, truly a "black technology" in the field of 3D digital humans.

Application Scenarios: From Live Streaming to the Metaverse

The application potential of MNN TaoAvatar is extensive, and it has been validated in multiple internal scenarios at Alibaba. For example, 3D digital human technology has already been used to enhance user experience in live streaming and virtual events on platforms such as Taobao and Youku. Some typical application scenarios include:

E-commerce live streaming: By using lifelike 3D virtual hosts, MNN TaoAvatar can enhance user immersion while reducing labor costs.

Virtual social networking and meetings: Users can create personalized 3D virtual avatars to participate in virtual meetings or social interactions, enhancing immersive experiences.

Metaverse and AR: MNN TaoAvatar supports operation on AR devices (such as Apple Vision Pro), providing technical support for metaverse and VR applications.

Online education and entertainment: Through vivid virtual characters, MNN TaoAvatar can add fun and interactivity to educational and gaming content.

Notably, MNN TaoAvatar's low storage requirements and high compatibility make it especially suitable for mobile devices and AR equipment, laying a technical foundation for the popularization of the metaverse in the future.

Open Source Empowerment: Another Milestone in Alibaba's Digital Human Technology

The open sourcing of MNN TaoAvatar marks another important breakthrough in Alibaba's digital human technology domain. Previously, Alibaba's Tongyi Lab has launched digital human projects such as EchoMimic and OmniTalker, showcasing its deep accumulation in this field. The release of MNN TaoAvatar further strengthens the MNN ecosystem, providing global developers with convenient tools to explore 3D digital human applications.

The project address has been made public (https://github.com/alibaba/MNN), and developers can quickly get started through rich APIs and documentation, customizing their own 3D digital human applications. AIbase believes that the open source of MNN TaoAvatar not only lowers the development threshold for 3D digital human technology but will also accelerate its popularity in commercial scenarios, providing powerful technical support for content creators and enterprises.

For more details, please visit the project website: https://pixelai-team.github.io/TaoAvatar/ or the GitHub address: https://github.com/alibaba/MNN. [] (https://ai-bot.cn/taoavatar/) [](https://www.aitop100.cn/infomation/details/26939.html)