SHUG 11. Anomaly Detection - Ted Dunning

Title: Anomaly Detection

Speaker: Ted Dunning

Abstract:

The basic ideas of anomaly detection are simple. You build a model and you look for data points that don’t match that model. Building a practical anomaly detection system requires deal with practical details starting with algorithm selection, data flow architecture, anomaly alerting, user interfaces and visualizations. We will describe the major classes of anomaly detection systems and show how to build anomaly detection systems for:

a) rate shifts to determine when events such as web traffic, purchases or process progress beacons shift rate

b) topic spotting to determine when new topics appear in a content stream such as Twitter

c) network flow anomalies to determine when systems with defined inputs and outputs act strangely.

While describing how to solve these problems, we will describe how clustering, dimensionality reduction, and density estimation can be used in systems that adapt and learn about their environment and how these systems can tell you when something has changed.

Bio:

Ted Dunning, Chief Application Architect @ MapR

Ted has held Chief Scientist positions at Veoh Networks, ID Analytics and at MusicMatch (now Yahoo Music). Ted is responsible for building the most advanced identity theft detection system on the planet, as well as one of the largest peer-assisted video distribution systems and ground-breaking music and video recommendations systems.

Ted has 15 issued and 15 pending patents and contributes to several Apache open source projects including Hadoop, Zookeeper and Hbase™. He is also a committer for Apache Mahout. Ted earned a BS degree in electrical engineering from the University of Colorado; a MS degree in computer science from New Mexico State University; and a Ph.D. in computing science from Sheffield University in the United Kingdom. Ted also bought the drinks at one of the very first Hadoop User Group meetings.

-------------------------------------------------

Additional information

RSVP to the meetup

Please RSVP to this meetup, since we need to put everybody on a guest list for entering the Spotify office. The event will be held in the cafeteria of the Spotify office, so don’t go to the normal entrance but to the 11th floor.

Pizza and drinks

Thanks to Spotify, pizza and beverages will be available for the participants during the meetup. This is another reason to RSVP to this meetup, if you are willing to come - it will help us to estimate the number of pizzas and drinks based on declared attendance.

The entrance

The door will be open between 17:45 and 18:15* Because of fire regulations, we need to keep a list of everybody in the building, so please make sure that you get your name ticked off the list at the entrance or (in case of a +1), make sure that the person at the door puts your name on the list.

*Unfortunately we can not leave the door open all the time (the company security policy), nor have a person that will be constantly watching for guests coming late. If you need to come later, please let us know in the comments below, so that somebody will come to the door to open it a given time.

See you at soon!

SHUG Team