Rare and Hidden Event Detection

This is a past event

210 people went


For our June Meetup, we are thrilled to have Gerhard Pilcher from Elder Research, Inc (http://www.datamininglab.com/)., presenting on the topic of rare event detection in data sets.

6:30pm -- Networking and Refreshments 7:00pm -- Introduction 7:10pm -- Mr. Pilcher's presentation and Q&A around 8:30pm -- Adjourn for Data Drinks Abstract

There are a collection of interesting problems associated with rare and hidden events. Most people naturally associate this class of problems with fraud detection and criminal activity but there are many other applications in science and marketing. By definition there are a lot of “non-events” in Rare Event detection resulting in noise that confuses attempts to model or discriminate among event outcomes. I will share some of our experiences with sorting through the noise and amplifying the response signal and discuss some techniques to increase confidence (or not!) in the resulting model. I use the term “hidden event”, also called unsupervised modeling, to describe a set of problems where the event outcome is unknown or the number of known cases is too small to be useful in machine learning algorithms. Detecting hidden events requires a much higher level of subject matter knowledge. I will discuss some general approaches to this class of problem and then focus on Mahalanobis distance measurement as an example technique for anomaly detection. I hope everyone will have the opportunity to add a few new tools to their data analysis “tool box”.


Gerhard Pilcher's work experience spans both private and government sectors, and has featured applications of data mining techniques to Fraud Detection and Risk Management. He currently serves as Vice President and Senior Scientist at Elder Research. Among his previous roles, he was Chief Technology Officer and VP of Engineering for Pulse Communications, where he directed the design of early digital subscriber line (DSL) systems.

Gerhard has served on various boards including the Strategic Advisory Board for the North Carolina State University Computer Science Department. He has a Master of Science in Analytics (Institute for Advanced Analytics, NCSU) and Bachelor of Science in Computer Science from NCSU.