Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

Bay Area Research in NLP and ML
Bay Area Research in NLP and ML
Public group

Mya Systems

27 Maiden Lane 6th Floor · San Francisco, ca

How to find us

If you are registered, there will be someone to meet you at the ground floor of 27 Maiden Lane who will let you up to the 6th floor between 6:30 and 7:00PM. DOORS WILL CLOSE AT 7:00PM SHARP. If you arrive after 7:00PM, you will not be able to enter!

Location image of event venue


Urvashi Khandelwal (NLP Group, Stanford University) will give a talk on the role that linguistic context plays in the behavior of neural language models. Food and drinks will be provided when the doors open at 6:30PM, doors close at 7:00PM sharp, and the talk begins at 7:00PM. No attendees will be able to enter the building after 7:00PM, so arrive early! Interested in presenting your work at a future Bay Area Research in NLP and ML Meetup? Click here:

Abstract of Urvashi's talk:
We know very little about how neural language models (LM) use prior linguistic context. In this paper, we investigate the role of context in an LSTM LM, through ablation studies. Specifically, we analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped. On two standard datasets, Penn Treebank and WikiText-2, we find that the model is capable of using about 200 tokens of context on average, but sharply distinguishes nearby context (recent 50 tokens) from the distant history. The model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context (beyond 50 tokens), suggesting the distant past is modeled only as a rough semantic field or topic. We further find that the neural caching model (Grave et al., 2017b) especially helps the LSTM to copy words from within this distant context. Overall, our analysis not only provides a better understanding of how neural LMs use their context, but also sheds light on recent success from cache-based models.

About Urvashi:
Urvashi Khandelwal is a fourth year PhD student in Computer Science at the Stanford NLP Group, where she is advised by Professor Dan Jurafsky. Her research interests lie at the intersection of natural language processing and machine learning. More specifically, she is interested in analyzing and improving models of language generation to facilitate generalization.

About this Meetup group:
Organized by the AI team at Mya Systems, this Meetup group is looking to bring together people testing and iterating novel approaches for potential applications in Artificial Intelligence. Come network, listen to what folks are working on, and learn from interactive sessions with experts in NLP & ML.

About Mya Systems:
Founded in 2012, Mya Systems brings deep learning and NLP expertise together to disrupt the recruiting operational model. See more at