【新智元导读】
国内首个人人可用的国产Sora「清影」，已经引起了AI视频圈的轰动！才发布6天，生成视频数就已经破百万。并且，智谱AI也将同源的视频生产模型CogVideoX，一并开源了。
7月26日，智谱发布AI 生视频产品「清影」，30秒将任意文图生成视频，并上线在他们的AI助手「智谱清言」上，被誉为是国内首个人人可用的Sora。
就在发布6天后，「清影」生成到视频数便已突破了百万量级。
今天，智谱AI宣布将与「清影」同源的视频生成模型——CogVideoX正式开源。
团队表示，希望每一位开发者、每一家企业都能自由地开发属于自己的视频生成模型，从而推动整个行业的快速迭代与创新发展。
打开AI助手「智谱清言」即可体验
随着大型模型技术的持续发展，视频生成技术正逐步走向成熟。
以Sora、Gen-3等闭源视频生成模型为代表的技术，正在重新定义行业的未来格局。
然而，截至目前，仍未有一个开源的视频生成模型，能够满足商业级应用的要求。
CogVideoX系列包含多个不同尺寸大小的开源模型。
目前已经开源的CogVideoX-2B，提示词上限为226个token，视频长度为6秒，帧率为8帧/秒，视频分辨率为720×480。
它在FP-16精度下的推理仅需18GB显存，微调则只需要40GB显存。这意味着单张4090显卡即可进行推理，而单张A6000显卡即可完成微调。
代码仓库：https://github.com/THUDM/CogVideo
模型下载：https://huggingface.co/THUDM/CogVideoX-2b
技术报告：
https://github.com/THUDM/CogVideo/blob/main/resources/CogVideoX.pdf
下面，我们就来看看CogVideoX生成的效果到底如何？
一个木制玩具船，在模拟海浪的蓝色地毯上航行，宛如在真的海水行驶一般。
A detailed wooden toy ship with intricately carved masts and sails is seen gliding smoothly over a plush, blue carpet that mimics the waves of the sea. The ship's hull is painted a rich brown, with tiny windows. The carpet, soft and textured, provides a perfect backdrop, resembling an oceanic expanse. Surrounding the ship are various other toys and children's items, hinting at a playful environment. The scene captures the innocence and imagination of childhood, with the toy ship's journey symbolizing endless adventures in a whimsical, indoor setting.
一辆白色越野车沿着松树环绕的陡峭土路快速行驶，可以看到车尾的尘土飞扬。
其实这个场景对于AI来说，相对较难，只有正确理解了物理世界，才不会让生成的尘土在车前扬起。
The camera follows behind a white vintage SUV with a black roof rack as it speeds up a steep dirt road surrounded by pine trees on a steep mountain slope, dust kicks up from it’s tires, the sunlight shines on the SUV as it speeds along the dirt road, casting a warm glow over the scene. The dirt road curves gently into the distance, with no other cars or vehicles in sight. The trees on either side of the road are redwoods, with patches of greenery scattered throughout. The car is seen from the rear following the curve with ease, making it seem as if it is on a rugged drive through the rugged terrain. The dirt road itself is surrounded by steep hills and mountains, with a clear blue sky above with wispy clouds.
下面这幅在战乱中，人物表情细节的刻画，细腻丰富。
In the haunting backdrop of a war-torn city, where ruins and crumbled walls tell a story of devastation, a poignant close-up frames a young girl. Her face is smudged with ash, a silent testament to the chaos around her. Her eyes glistening with a mix of sorrow and resilience, capturing the raw emotion of a world that has lost its innocence to the ravages of conflict.
针对更宏观的场景，CogVideoX也能将白雪皑皑的森林、无人穿梭的宁静小路，生动地描绘出来。
A snowy forest landscape with a dirt road running through it. The road is flanked by trees covered in snow, and the ground is also covered in snow. The sun is shining, creating a bright and serene atmosphere. The road appears to be empty, and there are no people or animals visible in the video. The style of the video is a natural landscape shot, with a focus on the beauty of the snowy forest and the peacefulness of the road.
舌尖上的美食，还得看国产AI视频模型。烧烤架上烤制的鸡肉和青椒烤串，让人看了垂涎欲滴。
Extreme close-up of chicken and green pepper kebabs grilling on a barbeque with flames. Shallow focus and light smoke. vivid colours
见识到CogVideoX惊艳视频生成能力，你一定非常好奇这是怎么做到的？
VAE
视频数据因包含空间和时间信息，其数据量和计算负担远超图像数据。
为应对此挑战，团队提出了基于3D变分自编码器（3D VAE）的视频压缩方法。
其中，3D VAE通过三维卷积同时压缩视频的空间和时间维度，实现了更高的压缩率和更好的重建质量。
模型结构包括编码器、解码器和潜在空间正则化器，并通过四个阶段的下采样和上采样实现压缩。
时间因果卷积确保了信息的因果性，减少了通信开销。而上下文并行技术的采用，则可以适应大规模视频处理。
实验中，团队发现大分辨率编码易于泛化，而增加帧数则挑战较大。
- 然后通过上下文并行在更高帧率上进行微调。
训练损失函数结合了L2损失、LPIPS感知损失和3D判别器的GAN损失。
专家Transformer
团队使用VAE的编码器将视频压缩至潜在空间，然后将潜在空间分割成块并展开成长的序列嵌入z_vision。
同时，使用T5将文本输入编码为文本嵌入z_text，然后将z_text和z_vision沿序列维度拼接。拼接后的嵌入被送入专家Transformer块堆栈中处理。
最后，反向拼接嵌入来恢复原始潜在空间形状，并使用VAE进行解码以重建视频。
