Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies

2026-07-08 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Advanced, quick

Summary

The study evaluates sequence-based neural networks (LSTM, GRU) for classifying autism-related self-stimulatory hand behaviors from video, addressing challenges in architecture, temporal sampling, and data augmentation for small datasets. Using the Self-Stimulatory Behavior Diagnosis (SSBD) dataset, LSTM and GRU models achieved peak accuracies of 97.5% and 98.75% respectively, at a 15-frame sampling interval, significantly outperforming prior convolutional neural network (CNN) baselines (62-76% accuracy). The research also explored ten data augmentation strategies for an I3D transfer learning pipeline. Horizontal flip yielded the highest standalone accuracy (48.78%), and upsampling proved critical, as its exclusion caused the largest performance degradation. A personalized machine learning approach, training per-subject models on temporally split video segments, showed consistent predictions with a mean loss of 1.84 (SD 0.79).

Key takeaway

For Machine Learning Engineers developing remote behavioral screening tools for ASD, optimizing video frame sampling and augmentation is crucial. You should prioritize LSTM or GRU architectures, specifically targeting a 15-frame sampling interval, which achieved up to 98.75% accuracy. Ensure your data augmentation pipelines include upsampling, as its exclusion significantly degrades performance, and consider personalized models for consistent subject-specific predictions.

Key insights

Optimal frame rates and specific data augmentation significantly boost video-based autism behavior classification accuracy.

Principles

• LSTM/GRU models exceed CNN baselines.
• Optimal temporal sampling boosts accuracy.
• Upsampling is crucial for video augmentation.

Method

Trained LSTM and GRU models on pose-derived features from the SSBD dataset at various frame sampling intervals (1, 5, 15, 30, 45, 90 frames) and applied ten data augmentation strategies.

In practice

• Use 15-frame sampling for hand idiosyncrasy detection.
• Prioritize LSTM or GRU over CNNs.
• Include upsampling in video augmentation.

Topics

• Autism Spectrum Disorder
• Self-Stimulatory Behaviors
• Sequence Classification
• LSTM
• GRU
• Video Data Augmentation
• Frame Rate Optimization

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies — Sequence-based models with optimal temporal sampling significantly outperform CNNs for autism-related behavioral classification.
• SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation — SGMD accelerates video diffusion distillation by directly optimizing fake scores with stable Fisher objectives and dual potentials.
• Data Augmentation: A Fourier Analysis Perspective — Partial data augmentation can achieve full statistical benefits with significantly fewer samples than full group augmentation.
• LLM-Conditioned Synthesis of Pathological Gaits via Structured Gait-Language Representations — The framework synthesizes pathology-aware 3D gait data using LLMs and a specialized tokeniser, improving classification accuracy for scarce datasets.
• A machine learning study highlighting the challenges of fidgety movement recognition using vision and inertial sensors — Automated Fidgety Movement recognition in infants faces significant generalization challenges across unseen subjects.
• Active Subsampling for Measurement-Constrained M-Estimation of Individualized Thresholds with High-Dimensional Data — Active subsampling significantly accelerates individualized threshold estimation in high-dimensional, label-constrained settings.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.