Advancing Fair Facial Expression Recognition
Loading...
Authors
Kolahdouzi, Mojtaba
Date
Type
thesis
Language
eng
Keyword
Facial expression recognition , Group fairness , Fair machine learning
Alternative Title
Abstract
Facial expression recognition (FER) is a key component of affective computing, which enables machines to interpret human expressions. Despite significant advances, FER systems still face challenges related to both performance and fairness. This dissertation addresses these challenges through three key contributions.
First, we introduce FaceTopoNet, a novel end-to-end deep learning model that learns an effective tree topology over facial landmarks for robust expression recognition. By traversing the learned tree, the model generates a sequence that is fed into two streams: structure and texture. The former stream captures the geometric structure of face components and the latter encodes texture. These streams are fused via an attention mechanism to form the final representation. Extensive experiments on five benchmark datasets show that not only FaceTopoNet achieves state-of-the-art or competitive performance but also exhibits robustness against occlusions.
Second, to enhance the fairness of FER models, we propose a novel loss function, named kernel mean shrinkage (KMS). KMS loss estimates the distributional difference between demographic groups among sensitive attributes in reproducing the kernel Hilbert space and minimizes it during training. Hence, it makes the sensitive attribute information less recognizable to the model. Furthermore, for the first time, we explore attractiveness as a sensitive attribute in faces and demonstrate that our proposed loss function is able to reduce such biases. Experimental results on CelebA and RAF-DB datasets demonstrate that our method improves fairness of FER models.
Third, we explore the impact of training optimizers on fairness, a notion that has been generally overlooked in this area. Using stochastic differential equation analysis, we show that adaptive optimizers like RMSProp and Adam are more likely than stochastic gradient descent to converge to fairer solutions under data imbalance. This theoretical insight is supported by empirical results on three large-scale datasets, CelebA, FairFace, and MS-COCO, across multiple tasks and fairness metrics. Altogether, these contributions offer a comprehensive path toward designing FER systems that are not only more accurate but also fair in their predictions across diverse groups.