NeuroVFM: A Volumetric Medical Foundation Model Trained on 5 Million Brain Scans
Researchers from the University of Michigan introduced NeuroVFM, a neuroimaging visual foundation model trained on 5.24 million clinical MRI and CT scans. Powered by Vol-JEPA self-supervised learning, it operates directly on volumetric images in a latent space, outperforming GPT-5 on clinical triage tasks.
Impact: High
Why it matters
Most medical AI models rely on narrow, disease-specific labeling or noisy clinical reports, limiting their general applicability. NeuroVFM demonstrates that a self-supervised, vision-only architecture (Vol-JEPA) can learn robust representations of 3D brain scans directly from raw clinical data without any manual annotation, enabling highly efficient multi-task clinical decision support.
TL;DR
- 01NeuroVFM is a volumetric clinical foundation model trained on 5.24M uncurated MRI and CT volumes using self-supervised Vol-JEPA.
- 02By predicting representations in a latent space, the model eliminates the need for radiology reports or manual pixel-level reconstruction.
- 03It achieved 92.68 AUROC on CT and 92.49 on MRI across 156 diagnostic tasks, outperforming alternative learning methods.
- 04In prospective clinical triage testing, NeuroVFM-LLaVA outperformed GPT-5 with a 92.6% balanced accuracy versus 71.2%.
Key facts
- Pretraining dataset
- 5.24M MRI and CT volumes
- Diagnostic AUROC (CT)
- 92.68
- Diagnostic AUROC (MRI)
- 92.49
- Base encoder size
- 85.8M parameters
- Triage accuracy
- 92.6% (NeuroVFM-LLaVA) vs 71.2% (GPT-5)
- License
- MIT (Code) / CC-BY-NC-SA-4.0 (Weights)
Self-Supervised Volumetric Learning via Vol-JEPA
NeuroVFM avoids the bottleneck of paired radiology reports and disease-specific curation by using Vol-JEPA (Volume Joint Embedding Predictive Architecture). The model tokenizes 3D CT and MRI volumes into non-overlapping 4×16×16-voxel patches, splits them into a visible context and a masked target, and trains a student encoder to predict masked-region latents. A teacher encoder—maintained as an exponential moving average (EMA) of the student—generates ground-truth target latents. This design minimizes smooth L1 loss and focuses on shared neuroanatomical features rather than background artifacts.
Outstanding Benchmark Performance
Across 156 clinical diagnostic tasks (74 MRI and 82 CT), NeuroVFM reached 92.68 AUROC on CT and 92.49 on MRI. It consistently outperformed alternative learning objectives like voxel reconstruction (NeuroMAE) and report supervision (HLIP). In training efficiency, Vol-JEPA utilized fewer than 1,000 GPU hours, scaling over 7x faster than a 3DINO baseline and fitting 16x larger batches within the same memory limit.
Clinical Deployment and Constraints
In a prospective one-week clinical trial involving 1,155 patients, a LLaVA-style pipeline leveraging NeuroVFM and Qwen3-14B achieved a balanced triage accuracy of 92.6%, far exceeding GPT-5's 71.2%. However, its critical-finding sensitivity was 86.5%, meaning it missed 21 out of 155 critical findings. Consequently, the research team emphasizes using the tool for clinical decision support rather than autonomous diagnosis.
Try it in 2 minutes
from neurovfm import load_encoder, load_diagnostic_head, load_vlm, interpret_findings
encoder, preprocessor = load_encoder("mlinslab/neurovfm-encoder")
dx_head = load_diagnostic_head("mlinslab/neurovfm-dx-ct")
generator, gen_preproc = load_vlm("mlinslab/neurovfm-llm")
# Encode a study, then predict diagnoses
batch = preprocessor.load_study("/path/to/ct/study/", modality="ct")
embeddings = encoder.embed(batch) # [N_tokens, 768]
predictions = dx_head.predict(embeddings, batch) # [(label, prob, pred), ...]
# Generate preliminary findings, then optionally triage
vols = gen_preproc.load_study("/path/to/ct/study/", modality="ct")
findings = generator.generate(vols, clinical_context="LOC and nausea.")
triage = interpret_findings(findings, "LOC and nausea.", api_key="...")python
✓ When to use
- When performing medical image analysis on 3D CT and MRI brain scans.
- For clinical decision support and triage assistance to prioritize urgent cases.
- When building multi-modal medical applications such as report generation or visual question answering.
✕ When NOT to use
- Do not use as an autonomous primary screening tool without human-in-the-loop oversight.
- Avoid if strict FDA approval is currently required for deployment.
- Avoid when 100% critical-finding detection is required without human review (due to the 13.5% miss rate).
What to do today
- Explore the GitHub repository to test the volumetric MRI and CT diagnostic encoder pipeline.
- Review the paper to understand how Vol-JEPA outperforms language and voxel-reconstruction models.
Sources