Skip to content

Anomaly Detection: How to Artificially Increase your F1-Score with a Biased Evaluation Protocol

Anomaly detection is a widely explored domain in machine learning. Many models are proposed in the literature, and compared through different metrics measured on various datasets.
The most popular metrics used to compare performances are F1-score, AUC and AVPR.
In this paper, we show that F1-score and AVPR are highly sensitive to the contamination rate.
One consequence is that it is possible to artificially increase their values by modifying the train-test split procedure.
This leads to misleading comparisons between algorithms in the literature, especially when the evaluation protocol is not well detailed.
Moreover, we show that the F1-score and the AVPR cannot be used to compare performances on different datasets as they do not reflect the intrinsic difficulty of modeling such data.
Based on these observations, we claim that F1-score and AVPR should not be used as metrics for anomaly detection. We recommend a generic evaluation procedure for unsupervised anomaly detection, including the use of other metrics such as the AUC, which are more robust to arbitrary choices in the evaluation protocol.

Damien Fourure*, Muhammad Usama Javaid*, Nicolas Posocco*, Simon Tihon*, Anomaly Detection: How to Artificially Increase your F1-Score with a Biased Evaluation Protocol, In Proc. of The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2021.

* equal contributions

Click here to access the paper.

Releated Posts

Muppet: A Modular and Constructive Decomposition for Perturbation-based Explanation Methods

The topic of explainable AI has recently received attention driven by a growing awareness of the need for transparent and accountable AI. In this paper, we propose a novel methodology to decompose any state-of-the-art perturbation-based explainability approach into four blocks. In addition, we provide Muppet: an open-source Python library for explainable AI.
Read More

Insights from GTC Paris 2025

Among the NVIDIA GTC Paris crowd was our CTO Sabri Skhiri, and from quantum computing breakthroughs to the full-stack AI advancements powering industrial digital twins and robotics, there is a lot to share! Explore with Sabri GTC 2025 trends, keynotes, and what it means for businesses looking to innovate.
Read More