Estimating Expected Calibration Errors

Uncertainty in probabilistic classifiers predictions is a key concern when models are used to support human decision making, in broader probabilistic pipelines or when sensitive automatic decisions have to be taken.
Studies have shown that most models are not intrinsically well calibrated, meaning that their decision scores are not consistent.

Continue reading

A Survey of Maturity Models in Data Management

Maturity models are helpful business tools that refine and develop how organizations conduct their businesses and benchmark their maturity status against a scale or with industry peers. They serve to prioritize the actions for improvement better and control the progress in reaching the target maturity stage.

Continue reading

Towards a Continuous Evaluation of Calibration

For safety-critical systems involving AI components (such as in planes, cars, or healthcare), safety and associated certification tasks are one of the main challenges, which can become costly and difficult to address.

One key aspect is to ensure that the decisions a machine-learning classifier makes are properly calibrated.

Continue reading