What weâre about
DEVDAY is a monthly informal event for developers to share their experiences, ideas, opinions & perspectives about technology.
Upcoming events (1)
See all- Democratizing Large Model Training on Smaller GPUs with FSDPSahaj AI Software Pvt. Ltd., Bengaluru
đď¸ Please RSVP here to get your free entry pass for the event
Are you struggling to train large AI models like LLMs and Vision Transformers because you don't have access to high-end, large-memory GPUs? You're not alone! This talk addresses that very challenge. Our primary goal is to make large-model training accessible to everyone, regardless of their hardware limitations. How will we achieve this? We'll dive deep into a powerful solution that combines the best of model architecture scaling (QLoRA) with advanced system scaling (Fully Sharded Data Parallel - FSDP). What will you learn and gain from attending?
- The Evolution of Scaling Techniques: Trace the progression of model design up to efficient techniques like Quantization and QLoRA. We'll also review the journey from various parallelism methods (data, model, pipeline) to the comprehensive solution that FSDP offers.
- A Deep Dive into FSDP: Understand the core fundamentals of Fully Sharded Data Parallel. Discover how it slashes memory usage by intelligently sharding model parameters, gradients, and optimizer states across all available devices while parallelizing your data.
- Practical Configuration and Best Practices: Get hands-on with key configuration techniques and PyTorch best practices. We'll cover essential strategies like mixed precision, CPU offloading, activation checkpointing, and optimal parameter wrapping.
- Analyzing the Trade-offs: Every powerful technique has its nuances. We'll analyze the inherent trade-offs of these methods, particularly how inter-GPU communication overhead impacts training time, showing you how to make large-scale training feasible even on smaller GPUs.
Who should attend?
This session is a must-attend for researchers and developers who are keen to train large models like LLMs on limited hardware. If you're curious about advanced scaling techniques in AI and deep learning, you'll find immense value in this talk!About the Speaker
Preethi Srinivasan
Solution Consultant, Sahaj Software
Preethi holds an M.S. (by Research) degree from IIT Mandi, where her thesis focused on medical image post-processing. She is the first author of publications in prestigious forums such as ACCV, WiML, and IEEE CBMS. At Sahaj, Preethi has developed cutting-edge ML prototypes, including systems for video understanding, LLM fine-tuning, and RAG-based QA. Her influential blog series on LoRA and intrinsic dimension has led to speaking engagements at prominent events like PyCon India 2024 and The Fifth Elephant 2025.
