Open-Source Video Generation: Diffusion & Transformer Models - AI Build & Learn

Welcome to AI Build & Learn, a weekly AI engineering stream where we pick a new topic and learn by building together.

• RSVP on the Luma to get direct live links and better reminders
• You can always find the live on YouTube: https://www.youtube.com/@sagecodes/streams

​This event is about generating video with AI, picking up where the image generation event left off. As with images, there's no single model we're committed to — we'll explore several and see how far the open-source options have come. Open source is the focus, but commercial models (Sora, Runway, and others) are fair game if you want to compare.

​Most current video models build on the same diffusion foundations as image generation, extended across time and increasingly using transformer/DiT backbones. We'll try a few open-source text-to-video and image-to-video models and talk through the practical tradeoffs: quality, clip length, speed, and hardware requirements.

​Some things to look up to get started:
​
Open-source models:

• ​Wan 2.2 (Alibaba Tongyi): versatile MoE model: text-to-video, image-to-video, and editing in one
• ​HunyuanVideo (Tencent): strong cinematic / photorealistic quality
• ​LTX-Video / LTX-2 (Lightricks): the fast one; ~5s clip in under a minute on a single GPU
• ​CogVideoX (Zhipu / THUDM): best at following detailed, multi-part prompts
• ​Mochi 1 (Genmo): flow-matching model known for fluid, coherent motion
• ​Stable Video Diffusion (Stability AI): earlier image-to-video, still widely used

​Tooling:

• ​Hugging Face Diffusers: video pipelines: https://github.com/huggingface/diffusers
• ​ComfyUI: node-based workflows (popular for video too): https://github.com/comfyanonymous/ComfyUI

​​​Resources

• ​​​GitHub: https://github.com/sagecodes/ai-build-and-learn
• ​​​Events Calendar: https://luma.com/ai-builders-and-learners
• ​​​Slack (Discuss during the week): https://slack.flyte.org/
• ​​​Hosted by Sage Elliott: https://www.linkedin.com/in/sageelliott/

​​In this stream

• Intro to topic
• ​​​​Community Discussion
• Practical examples

​​​Community challenge (optional)
​​​Try spending 30–90 minutes during the week learning or building something related to the topic, then share what you’re working on in Slack.

​​​Note on Flyte / Union
​​​You may see Flyte used in some demos. Flyte is an open-source AI orchestration platform maintained by Union (where I work) for building scalable, durable, and observable AI workflows. You do not need to use Flyte to participate.

• ​​​Union: https://www.union.ai/
• ​​​Flyte: https://flyte.org/

​​​Drop a comment with ideas for future topics (agents, RAG, MLOps, robotics, frameworks, and more).