Claimed “100% sensitivity and specificity in differentiating autistic individuals from typically developing controls using retinal photographs” . . . yeah, right.

2026-03-21 · Source: Statistical Modeling, Causal Inference, and Social Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, short

Summary

Two studies published in JAMA series journals claim exceptionally high diagnostic accuracy for autism spectrum disorder (ASD) using deep learning. The first study, published in 2023, reported 100% sensitivity and specificity in differentiating autistic individuals from typically developing controls using retinal photographs and deep learning algorithms, based on 1890 eyes from 958 participants. This contrasts sharply with the current gold standard (ADOS), which takes two hours and yields challenging diagnoses due to ASD's heterogeneity, or the Social Responsiveness Scale with ~0.85 sensitivity and ~0.75 specificity. The second study, from the same group, claimed near-perfect diagnostic accuracy (AUC > 0.99) using video recordings of a specific task. Concerns arise from the implausibility of 100% accuracy given ASD's spectrum nature and potential hidden confounding factors like camera differences or background brightness, especially since autistic participants were recruited from a single center while controls were retrospective from multiple centers.

Key takeaway

For AI scientists and research scientists evaluating diagnostic models, claims of 100% sensitivity and specificity for complex conditions like autism should trigger immediate skepticism. You should critically examine the methodology for potential hidden confounders, such as disparate data collection protocols between groups, and consider attempting to replicate the findings or analyze the publicly available data yourself. Such perfect metrics often indicate methodological flaws rather than groundbreaking success.

Key insights

Claims of 100% diagnostic accuracy for complex conditions like autism warrant extreme skepticism and rigorous scrutiny.

Principles

• Perfect diagnostic accuracy is highly improbable for heterogeneous conditions.
• Methodological differences can introduce spurious correlations.

In practice

• Investigate data availability and replication potential for high-accuracy claims.
• Scrutinize recruitment methods for potential confounding variables.

Topics

• Autism Diagnosis
• Deep Learning
• Retinal Imaging
• Diagnostic Accuracy
• Medical AI

Best for: AI Scientist, Research Scientist, AI Researcher, Data Scientist, AI Student

Related on AIssential

• Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text — Large language models poorly preserve diagnostic uncertainty in clinical text, posing a significant safety risk.
• Decision processes in 3D structural MRI schizophrenia classification evaluated with saliency maps — Explainable AI methods are crucial for validating deep learning models in medical diagnostics, revealing decision plausibility beyond performance metrics.
• Impact of Synthetic Lesional MR Images in Automated Focal Cortical Dysplasia Detection in Low-Data Scenarios — Synthetic FCD-MRIs generated by conditional networks can augment real data, reducing labeled data needs for automated detection.
• Looking Where the Eye Cannot See: Improving MS Diagnosis with Deep Learning — DeepMS improves MS diagnosis by analyzing subtle microstructural changes in normal-appearing white matter using deep learning.
• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• Physics-Informed Conditional Diffusion for Motion-Robust Retinal Temporal Laser Speckle Contrast Imaging — RetinaDiff uses physics-informed diffusion to reconstruct robust retinal blood flow from minimal speckle frames.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Statistical Modeling, Causal Inference, and Social Science.