Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies
Source: arXiv:2607.07957 · Published 2026-07-08 · By Raunak Mondal, Peter Washington
TL;DR
This paper tackles automated detection of autism-related self-stimulatory behaviors, focusing on two main challenges: selecting optimal sequence-based neural architectures and temporal sampling rates for classification of hand-flapping movements, and characterizing data augmentation strategies to improve training on the small Self-Stimulatory Behavior Diagnosis (SSBD) video dataset. Using pose-derived features from SSBD, the authors conducted systematic experiments comparing LSTM and GRU models at six frame sampling intervals, finding both outperform prior CNN baselines substantially, with peak accuracy of 98.75% (GRU) at sampling every 15 frames. They further evaluated ten data augmentation methods within an I3D transfer learning framework, identifying horizontal flip as the best standalone augmentation and upsampling as critical to pipeline performance through an ablation study. Additionally, a personalized machine learning approach, training per-subject models on temporally split video segments, yielded consistent predictions, suggesting within-video calibration potential. Overall, the paper offers concrete empirical guidance for model design, temporal resolution, and augmentation in clinical behavioral video classification tasks constrained by limited data.
Key findings
- LSTM and GRU models achieve peak accuracies of 97.5% and 98.75% respectively on binary hand-flapping classification from SSBD, substantially exceeding prior CNN baselines of 62–76% accuracy.
- Both sequential architectures peak at a temporal sampling interval of every 15 frames, with performance degrading at both higher and lower sampling rates.
- At native frame rate (1 frame), LSTM accuracy is 90.0%, GRU accuracy 92.5%, indicating overfitting/redundancy; at extreme downsampling (90 frames), accuracy drops to ~81% for both.
- Horizontal flip augmentation yields highest standalone accuracy (48.78%) on the three-class SSBD task in an I3D transfer learning pipeline, outperforming all other individual augmentations.
- Excluding upsampling from the full augmentation pipeline causes the largest performance degradation (training loss 5.0101 vs lower losses for other excluded augmentations), making it the most critical augmentation.
- Spatial augmentations outperform temporal augmentations for this behavioral video classification task, as temporal modifications tend to distort diagnostically relevant motion patterns.
- The personalized machine learning approach, training per-subject models on 80-20 temporal splits within videos, produces consistent predictions with mean loss 1.84 and standard deviation 0.79 across subjects.
- GRU model offers the highest single accuracy (98.75%) with fewer parameters and faster training compared to LSTM, favoring its use in resource-constrained applications.
Threat model
The adversary is an automated classifier operating on video recordings to detect autism-related self-stimulatory behaviors; no explicit adversarial attacks, obfuscation, or evasion attempts are considered. The model assumes access to video frames or pose features but cannot manipulate or spoof input data.
Methodology — deep read
The paper investigates two core questions on the SSBD dataset: architecture and frame sampling interval for sequence classification, and effective data augmentation for transfer learning on small behavioral videos. The threat model implicitly assumes an automated classifier attempting to identify self-stimulatory behaviors from video without adversarial interference.
Data provenance: The SSBD dataset contains 75 publicly sourced annotated videos averaging 90 seconds each, labeled for three behavior classes (hand flapping, head banging, spinning). For architecture evaluation, a subset of 50 hand-flapping videos with 90 frames each was used. Pose-derived geometric features were extracted per frame using body keypoint detection pipelines.
Experiment 1: Sequence-based LSTM and GRU models with one recurrent layer (LSTM or GRU), followed by dropout (p=0.5) and softmax dense output layer, were trained to classify hand flapping vs non-flapping. Inputs were sampled at six intervals (frames 1,5,15,30,45,90), yielding sequence length = floor(90/k). Models were trained using categorical cross-entropy and the Adam optimizer. An 80-20 train-test split was used. Metrics recorded included accuracy, precision, recall, and loss. No cross-validation or statistical tests were performed.
Experiment 2: The entire SSBD was used for 3-class behavior classification via a transfer learning pipeline incorporating I3D pretrained on Kinetics-400 with RGB and optical flow dual streams. The I3D backbone provided spatiotemporal feature extraction followed by dense layers with tanh and softmax activations. Ten data augmentation techniques were evaluated individually (standalone protocol) and by leave-one-out ablation within the full augmentation pipeline across multiple trials. A personalized machine learning protocol was applied, splitting each video temporally into five segments (first 4 for training, last for testing) to measure within-video generalizability.
Training regimes were not extensively detailed regarding epochs, hardware, or hyperparameters beyond dropout and optimizer choice. Reproducibility is partially facilitated by use of public SSBD dataset and referencing known architectures; however, code and weight release details are unclear.
A concrete example: At 15-frame sampling, GRU achieved 98.75% accuracy on hand-flapping detection. Inputs consist of pose features downsampled every 15 frames, fed into a GRU layer with reset and update gates that compress temporal information, followed by dropout and classification layers. This method balances capturing repetitive temporal patterns while reducing frame redundancy, outperforming CNN methods that do not model sequences.
Evaluation focuses on accuracy metrics on hold-out splits; no cross-validation or deeper robustness testing (e.g., across video conditions or populations) was conducted. The augmentation ablation clearly isolates upsampling as critical, and standalone experiments highlight the relative merit of spatial versus temporal augmentations.
Technical innovations
- Systematic evaluation of LSTM versus GRU sequence models across multiple temporal sampling intervals for autism-related behavior classification from pose features, identifying 15-frame sampling as optimal.
- Comprehensive characterization of the impact of ten distinct data augmentation strategies on transfer learning performance in a clinical behavioral video domain, with an ablation study highlighting critical augmentation types.
- Introduction of a personalized machine learning protocol applying temporal partitioning within individual videos to support per-subject model calibration and consistent behavioral recognition.
- Demonstration that explicit temporal sequence modeling using LSTM/GRU substantially outperforms prior CNN-based spatial models on small, noisy behavioral video datasets.
Datasets
- Self-Stimulatory Behavior Diagnosis (SSBD) — 75 videos (~90 seconds each) — publicly sourced internet videos with caregiver consent
Baselines vs proposed
- CNN baseline (Khodatars et al.): accuracy = 62-76%, vs GRU proposed: peak 98.75% at 15-frame sampling
- Bag-of-words SVM (original SSBD baseline): accuracy = 47.1%, vs horizontal flip augmentation standalone: 48.78%
- LSTM at 15 frames: accuracy 97.5% vs GRU at 15 frames: 98.75%
- At 1-frame sampling: LSTM 90.0%, GRU 92.5%; at 90-frame sampling: both ~81%
Limitations
- Small dataset size (50-75 videos) restricts generalizability and statistical confidence; no cross-validation or confidence intervals computed.
- Dataset heterogeneity due to publicly sourced videos introduces uncontrolled variability in recording conditions affecting model robustness.
- Absence of statistical significance testing limits confidence that reported accuracy differences are meaningful.
- The personalized machine learning evaluation used only a single 80-20 temporal split per video without multiple splits or subject-level validation.
- Data augmentation experiments leverage I3D pretrained on general activity recognition (Kinetics-400), not clinical behavioral data, possibly limiting feature relevance.
- Evaluated only relatively simple sequential architectures (single LSTM/GRU layer); newer sequence models such as temporal convolutions or transformers not explored.
Open questions / follow-ons
- How would more advanced sequence models like transformers or temporal convolutional networks perform relative to LSTM/GRU in this domain?
- What is the robustness of these models under distributional shifts such as different video capture environments or subject populations?
- Can larger or more diverse clinical behavioral datasets improve transfer learning and augmentation design further?
- How much initial calibration data is minimally required for effective personalized machine learning in clinical deployment?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners focused on behavioral analysis, this study illustrates the importance of explicitly modeling temporal sequences rather than static frames to detect subtle, repetitive behaviors. The finding that sampling every 15 frames balances accuracy and computational cost provides practical insight for designing real-time, resource-efficient video classifiers. Additionally, the data augmentation analysis highlights that increasing training data quantity via upsampling can be more effective than applying temporal distortions, a useful consideration when dealing with small video datasets in security screening. Personalized models trained on temporally partitioned data suggest that adapting classifiers per user/session could enhance consistency in behavioral recognition tasks. These insights could inform bot behavioral pattern detection from video or sensor streams, especially where data scarcity and subtle temporal dynamics are challenges.
Cite
@article{arxiv2607_07957,
title={ Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies },
author={ Raunak Mondal and Peter Washington },
journal={arXiv preprint arXiv:2607.07957},
year={ 2026 },
url={https://arxiv.org/abs/2607.07957}
}