[PDG 487] Unlocking In-Context Learning for Natural Datasets Beyond LMs

Link to article: https://arxiv.org/pdf/2501.06256
Title: Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling
Content: The work studies why in-context learning emerges in autoregressive models, especially beyond text modalities where it is less automatic than in LLMs. It identifies exact token repetitions in training sequences and appropriate training-task difficulty as key factors that help models learn the mechanisms needed for stable ICL. Using these insights, the authors enable ICL on visual datasets and a harder EEG classification task.
Slack link: ml-ka.slack.com, channel: #pdg. Please join us -- if you cannot join, please message us here or to mlpaperdiscussiongroupka@gmail.com.

In the Paper Discussion Group (PDG) we discuss recent and fundamental papers in the area of machine learning on a weekly basis. If you are interested, please read the paper beforehand and join us for the discussion. If you have not fully understood the paper, you can still participate – everyone is welcome! You can join the discussion or simply listen in. The discussion is in German or English depending on the participants.