The Effect of Biased Training Regimes on the Calibration of Deep Neural Networks
Loading...
Authors
Obadinma, Stephen
Date
Type
thesis
Language
eng
Keyword
Machine Learning , Calibration , Deep Learning , AI , Classification
Alternative Title
Abstract
Deep learning has advanced significantly in the past decade. Being well-calibrated remains a key property that a model must possess for safe and practical deployment in many high-stakes domains. The confidence of a model's predictions must be representative of its predictive performance so that it is possible to conduct accurate failure prediction and downstream decision-making. Many recent high performing deep learning architectures suffer from undesirable levels of miscalibration, which necessitates adequately measuring and reducing this miscalibration across many potential scenarios. In this thesis, we critically examine the calibration issue from a novel perspective; namely how calibration properties are affected when the training regime prioritizes performance on certain subsets of the data distribution. A biased learning regime is commonly encountered and/or utilized in training classifiers, and leads to significant changes in how models learn a data distribution, and thus where they focus their performance and confidence. We look into this effect through two lenses where this occurs implicitly, with imbalanced data distributions, and explicitly, with curriculum learning. We show that for imbalanced data, current popular metrics hide a high degree of miscalibration on important minority classes. As a result we develop new metrics, chiefly contraharmonic expected calibration (CECE), that penalize high levels of miscalibration on minority classes. In addition we create a novel calibration method called weighted temperature scaling that prioritizes calibration on minority classes and empirically show its effectiveness compared to other calibration methods and imbalance techniques. Furthermore, we detail the factors affecting calibration on imbalanced data. In the case of curriculum learning, we find limited effects on calibration when utilizing it in standard training scenarios, but under label noise we show that can it can provide a strong benefit at better calibrating models while maintaining competitive predictive performance. We analyze the various factors that parameterize curriculum learning on their effect on model calibration, and demonstrate how their careful tuning is required to avoid miscalibration. Ultimately, we provide insights as to understanding how classifiers and their calibration properties function under different learning regimes, which we hope can be foundational in teaching machines to have more inherently calibrated learning processes.
Description
Citation
Publisher
License
Queen's University's Thesis/Dissertation Non-Exclusive License for Deposit to QSpace and Library and Archives Canada
ProQuest PhD and Master's Theses International Dissemination Agreement
Intellectual Property Guidelines at Queen's University
Copying and Preserving Your Thesis
This publication is made available by the authority of the copyright owner solely for the purpose of private study and research and may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner.
ProQuest PhD and Master's Theses International Dissemination Agreement
Intellectual Property Guidelines at Queen's University
Copying and Preserving Your Thesis
This publication is made available by the authority of the copyright owner solely for the purpose of private study and research and may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner.