Facial Emotion Recognition
Seven architectures on one clean face dataset: how much does each gain from tuning, and how small can the best get?
Methods
Overview
How much does each architecture gain from tuning?
And how far can the best go with longer training, late generalisation, and compression?
A FER2013 test face
10 of 10 annotators said happy.
Runs on your device; nothing is uploaded.Privacy noteYour photos and camera pictures stay on your device. “Upload” only opens a photo in this page: the networks run here in your browser, and nothing is sent anywhere. The website’s code is public, so anyone can check this.
From a course project to a proper study
The study began in 2024 as a course project with a fellow student: one DenseNet, tuned and compared with a plain CNN on FER2013. That project had serious flaws in training and validation: the DenseNet was trained on the test images, the hyperparameter search minimised the training loss, and the two networks were scored on different versions of the data.
In 2026 I picked it up again and rebuilt it from scratch, with what I had learned in the two years since. The dataset grew and was cleaned: FER2013 with its FER+ votes, merged with RAF-DB. Seven architectures instead of two, each with the same pipeline and the same tuning budget, every decision made on the validation set. Then the study went where it got interesting: tuning, grokking, and compression.
How the study ran
Data: two public datasets merged and cleaned into 44,296 faces. Architectures: seven, from a plain CNN to a vision transformer, all trained from scratch. Tuning: 30 Optuna trials each, then the network's shape for the three winners.
Grokking: the three winners trained far longer, then the setting where grokking was shown on images. Compression: four methods, alone and combined, against networks trained small from scratch.
The study rebuilds an earlier course project from scratch, after that project's evaluation turned out to be invalid. Nothing from it is reused.
- 01DataCan two datasets be merged into one clean set?44,296faces after cleaning; the rarest class grew from 81 to 718 training faces
- 02ArchitecturesWhich of seven architectures works best?85.1%test accuracy, ResNet-18; the gap between models shrinks from 13 to 7 points with tuning
- 03TuningHow much does each model gain from tuning?+0.4points at most for the CNNs with batch norm, against +3.0 to +6.6 for the others
- 04GrokkingDoes a network generalise late, long after memorising?+11points of validation accuracy, long after every training face was memorised; gradual, never a sudden jump
- 05CompressionHow much smaller can the winners get?16×smaller ResNet-18 at the same test accuracy (85.4% against 85.2%)
References
- Goodfellow et al. (2013): Challenges in representation learning: a report on three machine learning contests (FER2013)
- Barsoum et al. (2016): Training deep networks for facial expression recognition with crowd-sourced label distribution (FER+)
- Li, Deng & Du (2017): Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild (RAF-DB)
- Akiba et al. (2019): Optuna: a next-generation hyperparameter optimization framework
- Liu, Michaud & Tegmark (2023): Omnigrok: grokking beyond algorithmic data
- Zhang, Zou, He & Sun (2016): Accelerating very deep convolutional networks for classification and detection
Novelty scope: The datasets, architectures, and methods are established. The contribution is a controlled comparison: seven architectures with one pipeline, one dataset, and equal tuning budgets, guesses written before each step, and follow-ups on long training, grokking, and compression measured the same way.
Part 01 of 05Data
Can two noisy face datasets be merged into one clean, more balanced set?
FER2013 has 35,887 faces with FER+ labels: ten votes per face. RAF-DB adds faces from the web, and the most where FER2013 is thinnest: disgust grows from 81 to 718 training faces.
Cleaning
A face was dropped when the annotators disagreed, when it was a near-duplicate (perceptual hash), or when its label was not one of the seven emotions. Duplicates never cross the train, validation and test split. 44,296 faces remain.
One format
All faces are 48 × 48 pixels in grayscale. RAF-DB faces were cropped to match FER2013's looser framing.
Augmentation runs on the GPU: flips, small rotations and shifts, brightness and contrast, and random erasing.
References
- Goodfellow et al. (2013): Challenges in representation learning: a report on three machine learning contests
- Barsoum et al. (2016): Training deep networks for facial expression recognition with crowd-sourced label distribution
- Li, Deng & Du (2017): Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild
- Zauner (2010): Implementation and benchmarking of perceptual image hash functions · master’s thesis
- Zhong et al. (2020): Random erasing data augmentation
Part 02 of 05Architectures
Which architecture recognises emotions best, with its defaults and after tuning?
- default settings
- after tuning
- ImageNet-pretrained ResNet-18
Seven architectures, all trained from scratch on the same faces with the same pipeline. Batch norm decides the ranking: ResNet-18, VGG and DenseNet lead at about 85% and gain little from tuning. Tuning shrinks the spread between the models from 13 to 7 points.
Training curves
Each model trains 60 epochs with AdamW, a warmup, and a cosine decay. Validation accuracy settles in the last epochs, and the validation loss never turns up: no model overfits.
Validation accuracy
Validation loss
- ResNet-18
- VGG
- DenseNet
- CCT
- Simple CNN
- ConvNeXt
- ViT
Where the errors are
Disgust and fear are the hardest classes for every model. Both are rare, fear is often taken for surprise, and disgust for sadness or anger.
ResNet-18, tuned: true emotion (rows) against prediction
Recall per emotion, all seven tuned models
Size
DenseNet matches ResNet-18 with 14 times fewer parameters. An ImageNet-pretrained ResNet-18 is no better than the one trained from scratch.
References
- LeCun et al. (1998): Gradient-based learning applied to document recognition
- Simonyan & Zisserman (2015): Very deep convolutional networks for large-scale image recognition
- Ioffe & Szegedy (2015): Batch normalization: accelerating deep network training by reducing internal covariate shift
- He et al. (2016): Deep residual learning for image recognition
- Huang et al. (2017): Densely connected convolutional networks
- Dosovitskiy et al. (2021): An image is worth 16x16 words: transformers for image recognition at scale
- Hassani et al. (2021): Escaping the big data paradigm with compact transformers
- Liu et al. (2022): A ConvNet for the 2020s
- Loshchilov & Hutter (2019): Decoupled weight decay regularization
- Szegedy et al. (2016): Rethinking the Inception architecture for computer vision (label smoothing)
Part 03 of 05Tuning
How sensitive is each architecture to its settings?
Does the network's shape matter, once the settings are tuned?
- random start
- TPE
- default settings
Optuna tuned all seven with the same budget: 30 trials of 25 epochs, random first, then TPE, over eight settings such as learning rate, warmup, and class weights. The CNNs with batch norm score well almost anywhere; the others only in a narrow band.
What mattered
Warmup mattered most for four models, the learning rate for the two transformers, and class weighting for ResNet-18.
The shape
For the three winners, width, depth, and the number of stages joined the search, with 40 more trials each. The best shapes gained 0.2 points on the test set, within the noise. Smaller networks lost accuracy: the published sizes were already right.
- VGG
- ResNet-18
- DenseNet
References
- Akiba et al. (2019): Optuna: a next-generation hyperparameter optimization framework
- Bergstra et al. (2011): Algorithms for hyper-parameter optimization (TPE)
- Hutter, Hoos & Leyton-Brown (2014): An efficient approach for assessing hyperparameter importance (fANOVA)
- Jamieson & Talwalkar (2016): Non-stochastic best arm identification and hyperparameter optimization
Part 04 of 05Grokking
Does training far longer help?
Can a network generalise long after it has memorised its training set, as in grokking?
- normal start, train
- normal start, validation
- 3× start, train
- 3× start, validation
- 3× start, no weight decay
A simple CNN on 1,000 training faces, in the setting where grokking was shown on images: large initial weights, MSE loss, weight decay, and 100,000 steps. Validation accuracy rises long after training accuracy reaches 100%, but gradually. There is no sudden jump.
Longer training
The three winners trained 100 epochs at a constant learning rate, then cooled down. None beats its 60-epoch result by more than the noise.
VGG
ResNet-18
DenseNet
- tuned
- larger shape
- memorize (no regularisation)
Overfitting
Without augmentation, label smoothing, and dropout, training accuracy keeps climbing while validation accuracy peaks at epoch 18 and falls: the classic picture of overfitting.
Accuracy
- training faces
- validation
Loss
- training faces
- validation
Batch norm blocks it
VGG, trained on far past memorising, did not grok. With batch norm, weight decay cannot favour a simpler solution. It only raises the effective step size, until training breaks in spikes.
Training loss (log scale)
Validation accuracy
What moves with the late rise
Neural collapse falls tenfold during the late rise. This fits the view of grokking as a switch from lazy to rich learning: the large start first memorises with its random features, then learns new ones.
- normal start
- 3× start, weight decay 0.1
- 3× start, 0.01
- 3× start, none
Without batch norm
VGG without batch norm behaves like the simple CNN and ends lower. Its weight norm jumps about every 5,000 steps, the slingshot pattern of Adam. The jumps bring no gain.
Weight norm, relative to the usual start
Validation accuracy
References
- Power et al. (2022): Grokking: generalization beyond overfitting on small algorithmic datasets
- Liu, Michaud & Tegmark (2023): Omnigrok: grokking beyond algorithmic data
- Kumar et al. (2024): Grokking as the transition from lazy to rich training dynamics
- Papyan, Han & Donoho (2020): Prevalence of neural collapse during the terminal phase of deep learning training
- Lobacheva et al. (2021): On the periodic behavior of neural network training with batch normalization and weight decay
- van Laarhoven (2017): L2 regularization versus batch and weight normalization
- Thilak et al. (2022): The slingshot mechanism: an empirical study of adaptive optimizers and the grokking phenomenon
- Golechha (2024): Progress measures for grokking on real-world tasks
- Truong et al. (2026): Spectral entropy collapse as a phase transition in delayed generalisation
- Hägele et al. (2024): Scaling laws and compute-optimal training beyond fixed training durations
Part 05 of 05Compression
How much smaller and faster can the winners get without losing accuracy?
Which method works, and why?
- Output-based low rank
- Tucker-2
- Spatial split
- + int8
- Same size, from scratch
- uncompressed
Four methods, alone and combined: output-based low rank, Tucker-2, a spatial split of each 3 × 3 kernel, and int8. With ranks chosen per layer, a short repair, and int8, ResNet-18 becomes 16 times smaller at its original test accuracy.
Why it works
The weights are close to full rank, but the layer outputs are not. In VGG's last convolution, 7 of 616 directions hold 95% of the output variance. Methods that compress the outputs win.
- last conv, outputs
- last conv, weights
- early conv, outputs
- early conv, weights
Ranks per layer, then repair
One threshold for all layers breaks VGG: it cuts the last layers below the 6 directions that 7 classes need. Ranks chosen per layer from their measured effect avoid this, and 10 epochs of repair recover most of a deep cut.
- VGG
- ResNet-18
- DenseNet
VGG and DenseNet
VGG holds 4 and 8 times fewer parameters only with low rank. DenseNet, already small, holds 2 times and breaks in int8.
VGG
DenseNet
Speed depends on the hardware
On the GPU, every factorised model is slower for one face: more, thinner layers cost more launches. On one CPU thread, the closest to a browser, a compressed ResNet-18 is 6.8 times faster at 0.7 points less.
The fair baseline
A network trained small from scratch is as good for VGG and ResNet-18, and better for DenseNet. Compression saves a training run; it finds no network that training would miss.
- trained small from scratch
- compressed and repaired
References
- Zhang, Zou, He & Sun (2016): Accelerating very deep convolutional networks for classification and detection
- Kim et al. (2016): Compression of deep convolutional neural networks for fast and low power mobile applications
- Jaderberg, Vedaldi & Zisserman (2014): Speeding up convolutional neural networks with low rank expansions
- Tai et al. (2016): Convolutional neural networks with low-rank regularization
- Denton et al. (2014): Exploiting linear structure within convolutional networks for efficient evaluation
- Lebedev et al. (2015): Speeding-up convolutional neural networks using fine-tuned CP-decomposition
- Novikov et al. (2015): Tensorizing neural networks
- Eckart & Young (1936): The approximation of one matrix by another of lower rank
- Jacob et al. (2018): Quantization and training of neural networks for efficient integer-arithmetic-only inference
- Hinton, Vinyals & Dean (2015): Distilling the knowledge in a neural network
- Liu et al. (2019): Rethinking the value of network pruning