Flag an abnormal eye scan, and show where the problem is.
Jinhyung Ahn and a teammate combined a Vision Transformer, a VQ-VAE trained only on normal scans, and a small convolutional fusion module to detect retinal OCT anomalies (drusen, DME and CNV) and map where they are. The ViT classifier reached 97.31% accuracy on 2,000 held-out scans. The location maps were shown as examples, not scored.
In retinal optical coherence tomography (OCT), telling normal from abnormal scans is hard, and showing where an abnormality sits is harder. Earlier work paired a VQ-VAE with a slow autoregressive model. This team combined three parts instead. A Vision Transformer, trained on labeled scans, classifies normal versus abnormal and shows attention-rollout maps; on a held-out test set of 500 normal and 1,500 abnormal scans it reached 97.31% accuracy. A VQ-VAE trained only on 23,901 normal scans reconstructs each image, and the difference is shown as an alignment loss map. A small convolutional module then predicts VQ-VAE codes from ViT features, producing a negative log-likelihood map of unusual regions. The maps were presented as qualitative examples across drusen, DME and CNV; the team plans to benchmark them and to deepen the fusion module.
Keywords · retinal OCT · anomaly detection · Vision Transformer · VQ-VAE · explainability · medical imaging
Who did this work
- On this project
- Managed the technically demanding workflow and structured the research progress.
Students named here agreed to be featured. 1 other team member is not named. Role notes come from the Project Lead's feedback.
1Detection is not enough
In retinal OCT scans, abnormal features can be subtle, scattered, or look like normal anatomy. A model that only says "abnormal" gives a clinician little to check.
Earlier work used a VQ-VAE with an autoregressive model to score anomalies, but that approach is slow and mostly catches local reconstruction errors. The team wanted global detection and local maps together.
2Three parts working together
- Classified. Trained a Vision Transformer (256×256 input, 6 layers, 16 heads) on labeled scans and visualized gradient-weighted attention rollout.
- Reconstructed. Trained a VQ-VAE only on 23,901 normal scans, so abnormal regions reconstruct poorly; showed the gap as an alignment loss map.
- Fused. Trained a small convolutional prior to predict VQ-VAE codebook indices from ViT features, giving a negative log-likelihood map.
- Tested. Held out 500 normal and 1,500 abnormal scans covering drusen, DME and CNV.
3Strong detection, maps shown as examples
| Component | Output | How it was evaluated |
|---|---|---|
| Vision Transformer | Normal vs. abnormal + attention map | 97.31% test accuracy (2,000 scans) |
| VQ-VAE | Reconstruction + alignment loss map | Qualitative examples |
| Convolutional fusion | Negative log-likelihood map | Qualitative examples |
4Selected slides
Slides from the team's final presentation.
5What this does not show
- Anomaly maps were not compared with clinician annotations.
- No formal hyperparameter tuning was done.
- The fusion module is shallow; the team plans deeper versions.
- Speed was not measured.
6What this work shows
Researcher in Medical Imaging and Machine Learning
Senior Imaging Scientist at Novartis. MIT PhD in ultrasound imaging and medical robotics. Co-founded two medical startups.
The student ran the experiments and built the analysis. The Project Lead guided the research direction and reviewed the work.
Medical imaging AI · Machine learning research
A framework that pairs detection with explanation shows research maturity.
7What the Project Lead observed
The Project Lead wrote feedback on the project at the end of the program.
Written feedback only
This cohort ran before the four-level capability rating was recorded, so this page shows the Project Lead's written feedback without a level.
- Technical depth
- Exceptional technical depth and research maturity in combining VQ-VAE, Vision Transformers and saliency maps for drusen, DME and CNV.
- Foundations
- Jinhyung showed a strong command of the mathematical foundations of machine learning.
- Execution
- Managed a technically demanding workflow and structured research progress clearly, without prior applied research experience.
- Teamwork
- The Academic Coordinator noted her collaborative mindset, adaptability and positive leadership.
Source: the Project Lead's and Academic Team's written feedback, May 2025.
References
- Dosovitskiy, A. et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.
- van den Oord, A. et al. (2017). Neural discrete representation learning. NeurIPS.
- Jebril, H. et al. (2024). Anomaly detection in optical coherence tomography angiography (OCTA) with a vector-quantized variational auto-encoder (VQ-VAE). Bioengineering.
Join the AI and Computer Vision in BioTech – Novo Nordisk Project
Work with a medical imaging scientist on a real problem and leave with research you can explain.
