In this article Andrea Pugnana explores selective classification in machine learning, introducing a novel heuristic for improving classifier performance. He also discusses the challenges of performance metrics and highlights future research directions.
The predictive performance of classifiers is typically not homogeneous over the data distribution. This is a common issue in Machine Learning. In fact, identifying sub-populations with low performance could be helpful, e.g., for debugging and monitoring purposes, especially in high-risk scenarios.