Andrew Lampinen | Language models show human-like content effects on reasoning

Virtual London Machine Learning Meetup - 16.11.2022 @ 18:30 (UK)

Agenda:

• 18:25: Virtual doors open
• 18:30: Talk
• 19:10: Q&A session
• 19:30: Close

Sponsor: Evolution AI - Intelligent data extraction from corporate and financial documents.

Title: Language models show human-like content effects on reasoning

Speaker: Andrew Lampinen, Research Scientist at DeepMind

Paper: https://arxiv.org/abs/2207.07051

Abstract:
Abstract reasoning is a key ability for an intelligent system. Large language models achieve above-chance performance on abstract reasoning tasks, but exhibit many imperfections. However, human abstract reasoning is also imperfect, and depends on our knowledge and beliefs about the content of the reasoning problem. For example, humans reason much more reliably about logical rules that are grounded in everyday situations than arbitrary rules about abstract attributes.

The training experiences of language models similarly endow them with prior expectations that reflect human knowledge and beliefs. We therefore hypothesized that language models would show human-like content effects on abstract reasoning problems.

We will describe experiments testing this hypothesis across three logical reasoning tasks: natural language inference, judging the logical validity of syllogisms, and the Wason selection task (Wason, 1968). We find that state of the art large language models (with 7 or 70 billion parameters; Hoffman et al., 2022) reflect many of the same patterns observed in humans across these tasks -- like humans, models reason more effectively about believable situations than unrealistic or abstract ones. We will use these findings to reflect on some broader questions about reasoning processes, and comparing language models to humans.

Bio:
I am a Research Scientist at DeepMind. I focus on applying ideas from cognitive science to improve or analyze AI systems, especially in reinforcement learning and language. I completed his PhD at Stanford University, and his BA at UC Berkeley. In my spare time, I enjoy rock climbing.