In-Context Learning tricks: Longer Contexts, Better Models & Cheap Summarization

In-context learning is a very approachable way of getting large language models (LLMs) to 'learn' how to do your task. This month's talks will address this new wave of models, and how to use them effectively. We'll also hear about LLM pre-training for SEA languages.

"The How and Why of Longer Contexts" - Martin Andrews

Gemini 1.5 (now open to all) showed us what's possible with 1 million token context. But what kind of techniques are being used for long contexts, and why might extended context length be a game-changer?

"RWKV and Next Gen LLMs" - Eugene Cheah

While Eugene has been visiting SG (from SF), he has given mostly high-level talks about the RWKV project, and what the team has been up to. Hopefully, he'll be able to present a little more in-depth at our MeetUp than elsewhere - and also tell us about his company (https://recursal.ai/).

"Learnings from Pre-training an LLM Encoder for South East Asian languages from Scratch" - Srinivasan Nandakumar

Srinivasan will talk about his experiences of being on the team tasked with pre-training an SEA language encoder model - including the kind of challenges that were faced (and won't necessarily be discussed in any paper write-up).

"Creating a Cheap Fast Summarization & Note Taking System" - Sam Witteveen

In this talk Sam will go through building a set of summarization and note taking tools and agents. Focusing in on some of the tricks for getting long form summaries out of LLMs via using multiple calls and ICL manipulation.

***

Talks will start at 7:00 pm and end at 8:50pm or so, at which point people normally come up to the front for a bit of a chat with each other, and the speakers.

As always, we're actively looking for more speakers - both '30 minutes long-form', and lightning talks. For the lightning talks, we welcome folks to come and talk about something cool they've done with keras_core, TensorFlow, PyTorch, JAX and/or Deep Learning for 5-10mins (so, if you have slides, then #max=10). We believe that the key ingredient for the success of a Lightning Talk is simply the cool/interesting factor. It doesn't matter whether you're an expert or an enthusiastic beginner: Given the responses we have had to previous talks, we're sure there are lots of people who would be interested to hear what you've been playing with. If you're interested in talking, please just introduce yourself to Martin or Sam at one of the events.