Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation

Transcription

1 Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation Kendrick Boyd 1 Vítor Santos Costa 2 Jesse Davis 3 David Page 1 1 University of Wisconsin Madison 2 University of Porto, Portugal 3 KU Leuven, Belgium June 28, 2012 Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

2 Introduction 1 Unachievable Region 0.8 Precision Recall Unachievable Region Precision-recall curves cannot go through the unachievable region. Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

3 Introduction 1 Unachievable Region 1 Unachievable Region Precision Precision Recall Recall Unachievable region varies with the data set. Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

4 Introduction 1 Unachievable Region 1 Unachievable Region Precision Precision Recall Recall Unachievable region varies with the data set. Our Goal Understand and characterize the unachievable region. Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

5 Precision-Recall Analysis TP Precision = TP + FP TP Recall = TP + FN Actual P N P TP FP N FN TN pos neg Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

6 Precision-Recall Analysis TP Precision = TP + FP TP Recall = TP + FN Scores used in PR analysis Precision-recall (PR) curves Area under PR curve (AUCPR) F β Mean average precision Precision Actual P N P TP FP N FN TN pos neg Recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

7 Known PR Space Properties Precision has high variance at low recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

8 Known PR Space Properties Precision has high variance at low recall Can translate between PR curves and ROC curves 1 PR curve 1 ROC curve Precision True Positive Rate Recall False Positive Rate Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

9 Known PR Space Properties Precision has high variance at low recall Can translate between PR curves and ROC curves Class Skew Proportion of positive examples in a data set. Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

10 Known PR Space Properties Precision has high variance at low recall Can translate between PR curves and ROC curves ROC curves - independent of class skew PR curves - sensitive to class skew Class Skew Proportion of positive examples in a data set. Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

11 Outline 1 Introduction 2 Unachievable Points 3 Unachievable Region 4 Discussion Downsampling Aggregation F1 Score 5 Conclusion Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

12 Illustration of an Unachievable Point positive examples 200 negative examples Precision Hold precision constant at Recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

13 Illustration of an Unachievable Point positive examples 200 negative examples Precision Hold precision constant at Actual P N P N recall Recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

14 Illustration of an Unachievable Point positive examples 200 negative examples Precision Hold precision constant at Recall Actual P N P N recall Actual P N P N recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

15 Illustration of an Unachievable Point positive examples 200 negative examples Precision Hold precision constant at Recall Actual P N P N recall Actual P N P N recall Actual P N P N recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

16 Unachievable Points in PR Space Theorem Precision (p) and recall (r) must satisfy where π = pos pos+neg p is the class skew. πr 1 π + πr Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

17 Outline 1 Introduction 2 Unachievable Points 3 Unachievable Region 4 Discussion Downsampling Aggregation F1 Score 5 Conclusion Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

18 Unachievable Region Precision Recall Example PR curves with π = Unachievable Region Sample PR Curve Random Guessing Minimum PR Curve Random Guessing: randomly assigned labels according to class skew Minimum PR Curve: worst possible ranking Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

19 Minimum AUCPR Theorem The area of the unachievable region in PR space and the minimum AUCPR, for class skew π, is AUCPR MIN = 1 + (1 π) ln(1 π) π Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

20 Minimum AUCPR Theorem The area of the unachievable region in PR space and the minimum AUCPR, for class skew π, is (1 π) ln(1 π) AUCPR MIN = 1 + π AUCPR MIN π Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

21 Outline 1 Introduction 2 Unachievable Points 3 Unachievable Region 4 Discussion Downsampling Aggregation F1 Score 5 Conclusion Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

22 AUCPR Comparisons Unachievable region always included for free in AUCPR Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

23 AUCPR Comparisons Unachievable region always included for free in AUCPR Range of AUCPR scores is [AUCPR MIN, 1] and thus depends on skew Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

24 AUCPR Comparisons Unachievable region always included for free in AUCPR Range of AUCPR scores is [AUCPR MIN, 1] and thus depends on skew AUCPR not comparable from different skews Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

25 AUCNPR Normalized area under precision-recall curve AUCNPR = AUCPR AUCPR MIN 1 AUCPR MIN Pros Range of AUCNPR is [0, 1] regardless of skew With same skew, preserves ordering of AUCPR Cons No good interpretation as an area in PR space AUCNPR of random guessing is not simple Still sensitive to skew Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

26 Downsampling Often downsample positives and negatives differently Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

27 Downsampling Often downsample positives and negatives differently Changes unachievable region for PR analysis Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

28 Downsampling Often downsample positives and negatives differently Changes unachievable region for PR analysis Cohort Study Should preserve the true skew Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

29 Downsampling Often downsample positives and negatives differently Changes unachievable region for PR analysis Cohort Study Should preserve the true skew Case-control Study Artificially makes the ratio of negatives to positives 1:1, 2:1, etc. Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

30 Downsampling on Mammography Cohort Study Cohort DATA SET Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

31 Downsampling on Mammography Cohort Study Case-control Study Cohort Controls Cases DATA SET DATA SET Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

32 Downsampling on Mammography Original SAYU Radiologists Downsampled Negatives SAYU Radiologists Precision Precision Recall Recall 120:1 ratio (π = 0.008) 1:1 ratio (π = 0.5) AUCPR = AUCPR = Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

33 Downsampling on Mammography Original SAYU Radiologists Downsampled Negatives SAYU Radiologists Precision Precision Recall Recall 120:1 ratio (π = 0.008) 1:1 ratio (π = 0.5) AUCPR = AUCPR = AUCPR MIN = AUCPR MIN = AUCNPR = AUCNPR = Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

34 Aggregating Results Sometimes want to combine results from problems with different skews Cross-validation folds Multiple tasks Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

35 Aggregating Results Sometimes want to combine results from problems with different skews Cross-validation folds Multiple tasks AUCNPR is a step in the right direction AUCNPR range is [0,1] for each fold/task Mean AUCNPR gives mean fraction of achievable area obtained But more work is needed! Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

36 F1 Score and Unachievable Region F1=0.8 F1 Contours Precision F1=0.6 F1 = 2pr p+r Recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

37 F1 Score and Unachievable Region A F1=0.8 Unachievable Region Minimum PR Curve F1 Contours Precision B F1=0.6 F1 = 2pr p+r Recall Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

38 Outline 1 Introduction 2 Unachievable Points 3 Unachievable Region 4 Discussion Downsampling Aggregation F1 Score 5 Conclusion Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

39 Conclusions Awareness of the unachievable region is critical for precision-recall analysis Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

40 Conclusions Awareness of the unachievable region is critical for precision-recall analysis Shown how to compute this region from just the class skew Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

41 Conclusions Awareness of the unachievable region is critical for precision-recall analysis Shown how to compute this region from just the class skew Recommendation: Always show unachievable region in figures Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

42 Conclusions Awareness of the unachievable region is critical for precision-recall analysis Shown how to compute this region from just the class skew Recommendation: Always show unachievable region in figures AUCNPR: a first step towards scores that account for the unachievable region Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

43 Acknowledgments CIBM Training Program (NIH 5T15LM007359) NIGMS grant R01GM NLM grant R01LM NIEHS grant 4R01ES Elizabeth S. Burnside Jude Shavlik ERDF through the Programme COMPETE Portuguese Government through FCT Project HORUS ref. PTDC/EIA-EIA/100897/2008 European Union Seventh Framework Programme FP7/ Research Fund K.U. Leuven (CREA/11/015 and OT/11/051) EU FP7 Marie Curie Farerr Integration Grant (#294068) FWO-Vlaanderen (G ) UW Carbone Cancer Center Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

44 Thank you Questions? Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

45 Modifying F1 Score A few desirable properties for a modified F1 score, f, f (r, p) = 0 if p = f (r 1, p) < f (r 2, p) iff r 1 < r 2 f (r, p 1 ) < f (r, p 2 ) iff p 1 < p 2 rπ 1 π + rπ Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

46 Modifying F1 Score A few desirable properties for a modified F1 score, f, f (r, p) = 0 if p = f (r 1, p) < f (r 2, p) iff r 1 < r 2 f (r, p 1 ) < f (r, p 2 ) iff p 1 < p 2 rπ 1 π + rπ Impossible! 0 = f (0, 0) < f (0, π) < f (1, π) = 0 Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

47 Modifying F1 Score Relaxed properties for a modified F1 score, f, f (r, p) = 0 if p = f (r 1, p) f (r 2, p) iff r 1 r 2 f (r, p 1 ) f (r, p 2 ) iff p 1 p 2 rπ 1 π + rπ Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

48 Modifying F1 Score Relaxed properties for a modified F1 score, f, f (r, p) = 0 if p = f (r 1, p) f (r 2, p) iff r 1 r 2 f (r, p 1 ) f (r, p 2 ) iff p 1 p 2 rπ 1 π + rπ One possible f f (r, p) = { 0 if p π 2(p π)r p π+(1 π)r if p > π Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

49 Proof of Unachievable Points Theorem Theorem Precision (p) and recall (r) must satisfy where π = pos pos+neg p πr 1 π + πr is the proportion of positive examples. TP p = TP + FP TP TP + (1 π)n rπn = rπn + (1 π)n rπ = rπ + (1 π) Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

50 Proof of Minimum AUCPR Theorem Theorem The area of the unachievable region in PR space and the minimum AUCPR, for proportion of positives π, is AUCPR MIN = 1 + (1 π) ln(1 π) π AUCPR MIN = = 1 0 rπ 1 π + rπ dr rπ + (π 1) ln(π(r 1) + 1) π r=1 r=0 = 1 (π + (π 1)(ln(1) ln(1 π))) π (1 π) ln(1 π) = 1 + π Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,

51 Proof of Minimum AP Theorem Theorem The minimum average precision, for pos and neg positive and negative examples, respectively, is AP MIN = 1 pos pos i=1 i i + neg AP MIN = 1 pos = 1 pos = 1 pos pos i=1 pos i=1 pos i=1 πi pos 1 π + πi pos posi (pos+neg)pos 1 + pos pos+neg ( i pos 1) i pos+neg i+neg pos+neg = 1 pos pos i=1 i i + neg Boyd et al. (ICML 2012) Unachievable Region in PR Space June 28,