Skip to content

Estimating Expected Calibration Errors

Uncertainty in probabilistic classifiers predictions is a key concern when models are used to support human decision making, in broader probabilistic pipelines or when sensitive automatic decisions have to be taken.
Studies have shown that most models are not intrinsically well calibrated, meaning that their decision scores are not consistent with posterior probabilities.
Hence being able to calibrate these models, or enforce calibration while learning them, has regained interest in recent literature.
In this context, properly assessing calibration is paramount to quantify new contributions tackling calibration.
However, there is room for improvement for commonly used metrics and evaluation of calibration could benefit from deeper analyses.
Thus this paper focuses on the empirical evaluation of calibration metrics in the context of classification.
More specifically it evaluates different estimators of the Expected Calibration Error ($ECE$), amongst which legacy estimators and some novel ones, proposed in this paper.
We build an empirical procedure to quantify the quality of these $ECE$ estimators, and use it to decide which estimator should be used in practice for different settings.

Nicolas Posocco, Antoine Bonnefoy, Estimating Expected Calibration Errors, In Proc. of the The 30th International Conference on Artificial Neural Networks, 2021.

Watch the presentation on YouTube.

Click here to access the paper.

Releated Posts

Augment to Interpret: Unsupervised and Inherently Interpretable Graph Embeddings

In this paper, we study graph representation learning and show that data augmentation that preserves semantics can be learned and used to produce interpretations. Our framework, which we named INGENIOUS, creates inherently interpretable embeddings and eliminates the need for costly additional post-hoc analysis.
Read More

SANGEA: Scalable and Attributed Network Generation

In this paper, we present SANGEA, a sizeable synthetic graph generation framework that extends the applicability of any SGG to large graphs. By first splitting the large graph into communities, SANGEA trains one SGG per community, then links the community graphs back together to create a synthetic large graph.
Read More