Natural Language Processing for Investigative Journalism

This month we have Jonathan Stray presenting "Natural Language Processing for Investigative Journalism."


Journalists frequently have far too many documents to read manually, whether it's a 10,000 page response to a Freedom of Information Request or 250,000 leaked diplomatic cables. We've spent the last three years applying NLP and visualization techniques to this problem, building a system called Overview which has now been used by journalists all over the world. In this talk I'll show you exactly how Overview's language processing pipeline works. 

But I'll also talk about how we decided which algorithms to use and how to present the results to the user. Topic modeling is a powerful technique, but all such algorithms are derived by optimizing for statistical properties, not fitness to end-user tasks. We developed Overview through extensive collaboration with journalists and careful user testing, and the experience has taught us a great deal about the problem of making NLP results interpretable to users. 

Since Overview is open source, you can leverage our work to build your own user-friendly NLP applications with our plugin API.


Jonathan Stray is a computer scientist and a journalist. He is a Fellow at the Tow Center for Digital Journalism at Columbia University where he teaches computational journalism. He leads the Overview Project, an open source visualization system to help investigative journalists make sense of very large document sets. He has worked as an editor at the Associated Press, a freelance reporter in Hong Kong, and an algorithm designer for Adobe Systems.

Join or login to comment.

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy