We are excited to announce that the paper On Model Evaluation under Non-constant Class Imbalance by Jan Brabec, Tomáš Komárek, Vojtěch Franc and Lukáš Machlica got accepted to the 20th The International Conference on Computational Science (ICCS 2020). The authors investigate a common problem when real-world class-imbalance is different from the imbalance in the test set. This may lead to wrong conclusions about the industrial impact and suitability of proposed techniques. Therefore they introduce methods and toolboxes to tackle this problem.
Two of the authors, J. Brabec and T. Komárek, are affiliated with our department (the AI Center) and also work at the security company Cisco Systems, Inc. We are very proud to have our research presented at a conference rated A-rank according to the CORE classification.
On Model Evaluation under Non-constant Class Imbalance
Jan Brabec, Tomáš Komárek, Vojtěch Franc, Lukáš Machlica
Many real-world classification problems are significantly class-imbalanced to detriment of the class of interest. The standard set of proper evaluation metrics is well-known but the usual assumption is that the test dataset imbalance equals the real-world imbalance. In practice, this assumption is often broken for various reasons. The reported results are then often too optimistic and may lead to wrong conclusions about industrial impact and suitability of proposed techniques. We introduce methods focusing on evaluation under non-constant class imbalance. We show that not only the absolute values of commonly used metrics, but even the order of classifiers in relation to the evaluation metric used is affected by the change of the imbalance rate. Finally, we demonstrate that using subsampling in order to get a test dataset with class imbalance equal to the one observed in the wild is not necessary, and eventually can lead to significant errors in classifier's performance estimate. Read the paper here.
The International Conference on Computational Science is an annual conference that brings together researchers and scientists from mathematics and computer science as basic computing disciplines, researchers from various application areas who are pioneering computational methods in sciences such as physics, chemistry, life sciences, and engineering, as well as in arts and humanitarian fields, to discuss problems and solutions in the area, to identify new issues, and to shape future directions for research. The theme for ICCS 2020 is “20 Years of Computational Science”. More about the conference here.