Behind AI Large Model Training, a Data Industry Chain is Forming


On Sep 18, Chinese Internet Corpus 3.0 (120GB) was released at an AI security forum in Kunming, supporting AI development under central guidance.....
At a recent video cloud technology conference, Volcano Engine unveiled a significant innovation: a video preprocessing solution for large model training. This technology has been successfully applied to the Doubao video generation model, marking a significant advancement in AI video generation technology. Volcano Engine's President, Tan Dai, emphasized that AIGC and multimodal technology are profoundly changing user experience. Leveraging experience from Douyin, Volcano Engine is actively exploring the integration of AI large models with video technology to provide comprehensive solutions for enterprises. Wang Yue, Head of Video Architecture at Douyin Group, pointed out that large model training faces numerous challenges.
Tencent Cloud recently launched an upgraded version of the Xing Mai Network 2.0, aiming to enhance the efficiency of large-scale model training. In the previous version, the synchronization communication time for the computation results of large models accounted for over 50% of the total, resulting in low efficiency. The new version of Xing Mai Network 2.0 has been upgraded in several aspects:1. Supports networking of up to 100,000 cards in a single cluster, doubling the scale, with a 60% improv
ZTE has launched the latest AI server that supports large model training aimed at the training and inference needs of small to medium-sized models. ZTE has introduced the G5 series servers and plans to release the latest AI server supporting large model training this year.
Shanghai AI Lab has released the large model training toolbox XTuner, which supports various hardware adaptations, significantly lowering the training cost barrier. XTuner is compatible with multiple open-source large models and can execute tasks such as incremental pre-training and instruction fine-tuning. XTuner balances usability and configurability with a built-in standardized process for one-click training.