Data Leakage Inflates Accuracy in Alzheimer's MRI Deep Learning Benchmarks, Study Finds
Researchers at the Karpagam Academy of Higher Education report that data leakage and image duplication in popular Alzheimer's disease MRI datasets falsely inflate deep learning diagnostic accuracy.
Key points
- Patient-level data leakage found in 100% of test images under naive splitting for OASIS.
- Removing data leakage caused OASIS mean test accuracy to collapse from 100.0% to about 46%.
- Researchers released new leakage-free, patient- and duplicate-aware split manifests for both datasets.
A new analysis published in Nature Health Sciences reveals that artificial intelligence models trained on widely used Alzheimer's disease MRI datasets may rely on data memorization rather than true diagnostic capability. Researchers from the Department of Computer Science and Engineering at the Karpagam Academy of Higher Education in Coimbatore, India, evaluated two prominent Kaggle-hosted MRI collections: OASIS and the Kaggle-Augmented Alzheimer's Dataset (KAAD).
Flawed Splitting Methods Lead to Severe Overestimation
The investigation demonstrated that standard image-level random splitting fails to account for patient provenance, creating significant identity leakage between training and test sets. For the OASIS database, patient identifiers embedded in filenames showed that every single test image shared a patient with the training set across all 86,437 images. Meanwhile, exact-hash and perceptual-hash evaluations estimated that roughly one-fifth of KAAD images were near-duplicates.
When researchers removed this leakage, mean test accuracy plummeted from 100.0% down to approximately 46% for OASIS, and dropped from 68.0% down to a range of 58.9% to 62.8% for KAAD.
Furthermore, cross-dataset testing revealed that a classifier trained on OASIS performed entirely at a chance level of 25.0% when evaluated on KAAD, despite demonstrating perfect training accuracy. The authors noted that this study is a secondary, retrospective analysis of pre-existing, publicly available data and did not involve the collection of new human-subject data. What to know: while these findings suggest that much published accuracy reflects image memorization instead of a transferable diagnostic signal, readers should consult a qualified medical professional for any personal health decisions.
Source Nature Health Sciences — nature.com/articles/s41598-026-77108-1 ↗
Read next
🏃FitnessUnderstanding and Addressing Love Handles Through Lifestyle Changes
Experts clarify that spot reduction of fat is not physiologically possible, recommending a combination of calorie management and high-intensity interval training instead.
🏃FitnessCiele Releases New Woolmark-Certified Merino Winter Running Apparel
Montreal-based brand Ciele has launched its second collection of merino wool running gear, featuring performance-focused fabrics designed for cold-weather training.
❤️Conditions & CareBlood-Based Genetic Signature May Predict CAR T-Cell Therapy Success in Lymphoma
Researchers have identified a specific RNA signature in blood plasma that may help predict how well lymphoma patients respond to CAR T-cell therapy.
🧠AI AnalysisUltraprocessed Diets May Impact Brain Reward Systems in Younger Adults
A Virginia Tech study suggests that individuals aged 18 to 21 may experience changes in brain reward processing after consuming ultraprocessed foods, a response not obser…
🧠Mind & Mental HealthAddressing Body Image and the Reality of Gluteal Anatomy
A personal reflection on the social and psychological impact of body shape, specifically regarding the gluteal region, and the limitations of consumer solutions for perce…
❤️Conditions & CareMigraine and Smoking Linked to Increased Early Stroke Risk in Women
A large-scale study suggests that people who have given birth and experience migraines while also smoking face a significantly higher risk of stroke before age 60.