RankingBased Evaluation of Regression Models.pdf

Ranking-Based Evaluation of Regression Models Saharon Rosset, Claudia Perlich, Bianca Zadrozny IBM T. J. Watson Research Center P. O. Box 218 Yorktown Heights, NY 10598 {srosset, reisz, zadrozny}@ Abstract are likely to have high response from the ones which cor- respond to low response. An example is given in the case We suggest the use of ranking-based evaluation mea- study we present in Section 5, where the goal is to iden- sures for regression models, as a complement to the com- tify companies with high potential IT spending (IT Wal- monly used residual-based evaluation. We argue that in let). These are preferred targets for vigorous sales efforts some cases, such as the case study we present, ranking by IBM. If we assume that IBM’s sales resources are fixed, can be the main underlying goal in building a regression then identifying the largest IT Wallets, regardless of their model, and ranking performance is the correct evaluation actual numeric value, is the best support a regression model metric. However, even when ranking is not the contextu- can give. ally correct performance metric, the measures we explore Second, ranking-based measures are quite interpretable. still have significant advantages: They are robust against The two main evaluation methods we consider allow us to extreme outliers in the evaluation set; and they are inter- draw interpretations and connections: pretable. The two measures we consider correspond closely to