Nov BARUG Meeting: "ROC Day"

Agenda:
4:30 Networking
5:00 Announcements
5:05 Joseph Rickert - Some ROC Basics and History
5:20 Mario Inchiosa - ROC curves extended to multiclass classification, and how they do or do not map to the binary case.
5:45 Robert Horton - Six Ways to Think About ROC Curves
6:25 John Mount - How to Pick an Optimal Utility Threshold Using the ROC Plot
6:55 Nina Zumel - Squeezing the Most Utility from Your Models

#
J Rickert
Some ROC Basics and History
In this very brief introduction, I will provide a little background on the history of ROC curves and introduce the basics.

#
M Inchiosa
ROC curves extended to multiclass classification, and how they do or do not map to the binary case.

The ROC curve was originally developed for binary classification, but extensions of the ROC curve to multiclass classification are commonly used, as well. In fact, the R package “multiROC” implements such extensions. We will briefly discuss these extensions and consider the following question: if multiclass classification metrics can be used for problems with n classes, do they apply to the special case of binary classification, i.e. n=2?

#
R Horton
Six Ways to Think About ROC Curves
We will walk through six ways to understand ROC curves:

1. The discrete "Turtle's Eye" view, where labeled cases are sorted by score, and the path of the curve is determined by the order of positive and negative cases.

2. The categorical view, where we have to handle tied scores, or when scores put cases in sortable buckets.

3. The continuous view, where the cumulative distribution function (CDF) for the positive cases is plotted against the CDF for the negative cases.

4. The ROC curve can be thought of as the limit of the cumulative gain curve (or "Total Operating Characteristic" curve) as the prevalence of positive cases goes to zero.

5. The probabilistic view, where AUC is the probability that a randomly chosen positive case will have a higher score than a randomly chosen negative case.

6. The ROC curve emerges from a graphical interpretation of the Mann-Whitney Wilcoxon U Test Statistic, which illustrates how AUC relates to this commonly used non-parametric hypothesis test.

Once we're covered how to think about ROC curves, we'll look at some examples of using them to diagnose issues with machine learning classifiers

#
J Mount
How to Pick an Optimal Utility Threshold Using the ROC Plot

One of the classic applications of the ROC plot is to encode the efficient frontier of optimal specificity and sensitivity tradeoffs for a given signal detection problem. We will show the classic derivation of how to translate concrete business goals into the ROC framework and then use the ROC plot to pick off the optimal classification threshold for a given problem. I will emphasize discovering utility requirements through iterated negotiation.

A sneak peek of some of the points can be found here: https://win-vector.com/2020/10/10/how-to-pick-an-optimal-utility-threshold-using-the-roc-plot/

#
N Zumel
Squeezing the Most Utility from Your Models

The ROC concisely expresses all the sensitivity/specificity tradeoffs achievable by a given probability (or other score-returning) model. Generally, one converts such a model into a "hard" classifier by picking a decision threshold. A good threshold balances classifier precision/recall or sensitivity/specificity in a way that best meets the project or business needs. Unfortunately, the ROC is not the best visualization for picking the optimal threshold.

A more direct way to quantify and think about this optimal threshold is the notion of model utility, which maps the performance of a model to some notion of the value achieved by that performance. In this talke, we discuss how to estimate model utility and pick model thresholds for classification problems.

For an advance peek at some of these ideas see here: https://win-vector.com/2020/10/05/squeezing-the-most-utility-from-your-models/