Build LLMs From Scratch - AiA: Pre-Training LLMs
Details
Join our event-series to learn how to build Large Language Models (LLMs) from scratch.
More info here: Pre-Training LLMs
NEW: There is a full video for this chapter
*This event is sponsored by: Le Wagon Lisbon
If you didn't join the previous events, feel free to come to listen and interact.
***
Join us to build your LLM from scratch! Learn by doing together with a supportive community.
Full details: check out this page
***
## What
This is a hands-on learning bootcamp focusing on Large Language Models (LLMs), spanning several months. You will learn to design, pre-train, and fine-tune your own GPT-like model.
## Who
The program is suitable for people with a quantitative background. Working knowledge of Python, Pytorch, and Machine Learning is a plus but not mandatory.
Two main prerequisites are:
- be comfortable with computers & maths (bachelor level)
- willing to commit about 40h/month to focused learning
## How
- Study: read/watch materials, run the code, and solve the exercises
- Research: read papers, explore and experiment, try to break things
- Group meeting: monthly meetings, summarize key insights, Q&A
- Discord: discuss about anything, share resources, ask questions
## Wanna join?
IMPORTANT:
- Check out [program repo on Github]
- Please fill [Google Form] to join WhatsApp & Discord group
- Logistics: meetings are in-person, location & time will be updated in Discord & Meetup
See you there! Dan & Mark
***
# Program & Schedule
We plan to meet every 3/4 weeks, around the last week of the month.
- Month 0 Getting started - Mon, Nov 4
- Month 1 Tokenization & Embeddings - Mon, Nov 25
- Month 2 Project: build your own tokenizer - Mon, Jan 8
- Month 3 Attention Mechanisms - Mon, Feb 3rd
- Month 4 Transformer & GPT Architecture - Mon, Feb 24
- Month 5 Pre-training LLMs - Wed, Apr 2
- Month 6 Fine-tuning LLMs - Mon, May 5
- Month 7 Final Project: build your own GPT-2 - Mon, May 26
The first 3 months have optional materials allowing for everyone to acquire the fundamental knowledge required to build LLMs.
***
## About Us
Our goal is to democratize Machine Learning and AI. We experiment with hands-on projects on LLMs like RAGs, quantization, and other real-life applications. We believe that it does not matter who you are, where you come from, you can build and contribute to shaping the future with better and safer AI technologies.
***
## FAQs
1.Is there any fee?
No fee, no hidden cost, aside from the textbook. If you can't afford it, reach out to us. Most materials are publicly available.
2.What will I learn after completing the program?
At minimum, you'll gain a much deeper understanding of LLMs than from YouTube and blog-posts. At best, with the right resources, you'll spin out new custom LLMs every month.
3.How can I join the Discord and WhatsApp group?
Fill out the form in the event description to receive invitations to join the groups.
4.What kind of support is available?
Support is available in real-time on Discord. We may organize co-working days on top of the monthly meeting.
5.What if I miss a meeting?
Meeting discussions and resources will be available on Discord and GitHub to help you catch up.
6.Can I join after the series has started?
It's best to join from the start, but you can join later if you have the required time/knowledge to catch up.
Build LLMs From Scratch - AiA: Pre-Training LLMs