Measuring Anomalousness in Statistical Models

Measuring Anomalousness in Statistical Models
As the complexity of IT environments and the quantity of data people gather increases, proactively managing the health and security of these environments requires increasingly sophisticated monitoring tools. Rule based systems are either becoming unmanageable, i.e. in the case the setting of static thresholds to alert on unusual metric values, or in need of augmentation, i.e. in the case of identifying the signatures of known types of network intrusion attacks.

Many use cases in these areas fall under the umbrella of anomaly detection: for example, people want to know" is my system behaving differently to how it normally behaves", "is this IP address doing something different to others interacting with my system", "is this a rare log message", and so on. Due to the intrinsic randomness of the raw data, statistical models are ideally suited for problems in these areas. However, as mentioned, the complexity and the scale of the data poses significant challenges. Recent techniques from the fields of Data Mining, for example sketch data structures, Machine Learning, for pattern recognition, and accurate robust statistics, such M-estimators, proper characterisation of distribution tails, proper handling of heterogeneous data types and so on, are all useful for improving the quality and/or scalability of anomaly detection.

In this talk I aim to introduce the topic of anomaly detection for time series data, survey some of the data characteristics which make anomaly detection for real world problems challenging and describe some of the techniques we use for anomaly detection.

About the presenter:
Tom Veasey serves as Research Director at Prelert. Prior to working for Prelert Tom has worked as a consultant in a mathematical modelling consultancy, focusing primarily on satellite attitude and orbital control and phased array radar. He was a lead developer in the team responsible for Azuro's clock concurrent optimisation technology (subsequently acquired by Cadence) and has worked for a period on FX derivative pricing and risk management tools at Bloomberg LP. Tom holds a masters in physics from the University of Cambridge, UK.