Classifier EvaluationPresentation

1. Introduction

Radar ATR

Objectives

Method

Comparison

Metrics

2. Black Box Classifier

Inputs

Reference

Data to classify

Stages

Features

Scoring

Threshold

Selection

Declare

"Don't declare"

Outputs

Declaration

Unknown

"Don't Declare"

3. Evaluation Metrics

Probability of correct classification

Reliability

Probability of declaration

Forced declaration

Probability of false alarm

Probability of generalization

4. The Confusion Matrix

Test target matrix

Test variant target matrix

Test unknown target matrix

5. ROC Curve Analysis

Type (a) ROC curve

Type (b) ROC curve

Generating the curve

Interesting points in ROC space

6. Real World Examples

Introduce dataset

Pre-processing and classifier used

Impact of forced decision

Impact of a very cautious threshold

Using a target level of Pfa

Real world ROC curves

7. Summary

What have I told them about

8. What Comes Next?

How to use "Don't Declare" threshold

How to predict performance

Why do this?

Possibly use error bounds

Advert for my Australia paper