Robust event-based gesture recognition synthetic data augmentation
Talukder, Md Mahi Al Jubair (2026)
Kandidaatintyö
2026
School of Engineering Science, Laskennallinen tekniikka
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe20260622101687
https://urn.fi/URN:NBN:fi-fe20260622101687
Tiivistelmä
Event-based Dynamic Vision Sensors (DVS) offer low power consumption with high temporal resolution for gesture recognition. However, models trained on clean lab datasets, like DVS128 Gesture, suffer severe real-world performance degradation due to clutter, lighting, and viewpoint variations.
This thesis evaluates a pre-trained 3D Convolutional Neural Network (Gesture3DCNN). While achieving 93.9\% lab accuracy, performance dropped to 18.2\% on a separate real-world test set of 11 videos converted using the video-to-events (v2e) emulator. Heuristic filters failed, exposing a proposed ``Density Trap'' interpretation that misclassified background noise. To bridge this gap, synthetic domain randomization and fine-tuning were applied. Fine-tuning with 220 augmented single-subject training samples (generated from 11 separate videos) raised test accuracy to 72.7\% on 11 test videos. Expanding training to 440 samples across two subjects (from 22 separate videos) reduced accuracy to 63.6\% on 22 test videos. Further expanding to multi-view training data (1,320 samples from 66 separate videos) caused accuracy to drop to 30.3\% on 66 test videos.
Ultimately, domain randomization with real-world fine-tuning outperforms heuristic filtering, though severe multi-view generalization challenges persist.
This thesis evaluates a pre-trained 3D Convolutional Neural Network (Gesture3DCNN). While achieving 93.9\% lab accuracy, performance dropped to 18.2\% on a separate real-world test set of 11 videos converted using the video-to-events (v2e) emulator. Heuristic filters failed, exposing a proposed ``Density Trap'' interpretation that misclassified background noise. To bridge this gap, synthetic domain randomization and fine-tuning were applied. Fine-tuning with 220 augmented single-subject training samples (generated from 11 separate videos) raised test accuracy to 72.7\% on 11 test videos. Expanding training to 440 samples across two subjects (from 22 separate videos) reduced accuracy to 63.6\% on 22 test videos. Further expanding to multi-view training data (1,320 samples from 66 separate videos) caused accuracy to drop to 30.3\% on 66 test videos.
Ultimately, domain randomization with real-world fine-tuning outperforms heuristic filtering, though severe multi-view generalization challenges persist.