Does a GAN make a pneumonia classifier better? Here, it made it worse.
Sean Joe Lee compared five setups for detecting pneumonia on chest X-rays. A ResNet50 with traditional augmentation scored best: 0.993 AUROC, 0.889 specificity, 0.964 F1. Adding CycleGAN synthetic images lowered every metric, and the CycleGAN-based detectors fell to 0.53–0.65 AUROC. He then showed why: the GAN darkened images and changed bones and edges instead of lung tissue.
Generative models are often proposed to create extra training images for medical AI. This project tested whether that helps pneumonia detection on chest X-rays. Five setups were compared: a ResNet50 baseline with no augmentation, ResNet50 with traditional augmentation, ResNet50 with CycleGAN synthetic images plus traditional augmentation, and two CycleGAN-based detectors (an identity Normal→Normal mapping and a Normal→Pneumonia mapping). Images were preprocessed with CLAHE and training used focal loss. Traditional augmentation gave the best overall result (AUROC 0.9927, specificity 0.8889, F1 0.9639). Adding CycleGAN images lowered every metric slightly, and the CycleGAN-based detectors reached only 0.5298–0.6513 AUROC. Intensity analysis and Grad-CAM showed why: synthetic images were systematically darker, collapsed into a narrow intensity range, and changed ribs, heart borders and the spine more than the lung fields where pneumonia appears.
Keywords · pneumonia detection · chest X-ray · CycleGAN · data augmentation · Grad-CAM · failure analysis
Who did this work
- On this project
- Did the whole project alone: modeling, experiments and analysis.
Students named here agreed to be featured. Role notes come from the Project Lead's feedback.
1Why test GAN augmentation at all
Pneumonia affects hundreds of millions of people a year, and many countries have far fewer radiologists than recommended. Automated screening could help, and generative models are often proposed as a way to create more training images.
In deployed clinical pipelines, GAN-generated pathology is rarely used, partly because synthetic images can introduce subtle errors. This project tested whether CycleGAN images actually help, and if not, why.
2Five setups, same evaluation
- Preprocessed. Applied CLAHE contrast enhancement and trained with focal loss to handle class imbalance.
- Trained five setups. ResNet50 without augmentation; with traditional augmentation; with CycleGAN images plus traditional augmentation; a CycleGAN identity mapping; and a CycleGAN Normal→Pneumonia mapping.
- Measured the same way. Sensitivity, specificity, AUROC, F1 and precision for every setup.
- Looked inside. Compared intensity distributions of real and synthetic images, and used Grad-CAM to see where the models looked.
3Simpler won, and the failure has a mechanism
| Setup | Sensitivity | Specificity | AUROC | F1 |
|---|---|---|---|---|
| 1 · ResNet50, no augmentation | 1.000 | 0.769 | 0.993 | 0.902 |
| 2 · ResNet50 + traditional augmentation | 0.992 | 0.889 | 0.993 | 0.964 |
| 3 · + CycleGAN synthetic images | 0.972 | 0.882 | 0.980 | 0.951 |
| 4 · CycleGAN identity mapping | 0.927 | 0.257 | 0.651 | 0.782 |
| 5 · CycleGAN Normal→Pneumonia | 0.860 | 0.380 | 0.530 | 0.694 |
- Images got darker. Synthetic images shifted intensity down, by about 8% according to the analysis.
- Outputs collapsed. Synthetic intensities squeezed into a narrow range, losing the difference between normal and pneumonia.
- Wrong regions changed. The GAN altered ribs, heart borders and the spine most, and the central lung fields least.
4Selected slides
Slides from the final presentation.
5What this does not show
- Results come from one public chest X-ray dataset; no confidence intervals or repeated runs are reported.
- CycleGAN training was unstable; different settings might behave differently.
- The finding is about this setup, not proof that generative augmentation never helps.
6What this work shows
Researcher in Medical Imaging and Machine Learning
Senior Imaging Scientist at Novartis. MIT PhD in ultrasound imaging and medical robotics. Co-founded two medical startups.
The student ran the experiments and built the analysis. The Project Lead guided the research direction and reviewed the work.
Medical imaging AI · Machine learning research
A careful negative result with a clear mechanism is a strong research writing sample.
7What the Project Lead observed
The Project Lead wrote feedback on the project and rated the demonstrated capability on a four-level scale.
System-Level Insight
- 1Foundational Application
- 2Applied Reasoning
- 3Independent Problem Solving
- 4System-Level Insight
Level 4 of 4: anticipates broader implications, risks or downstream effects, and connects technical decisions to real-world impact.
- Framing the problem
- Challenged the assumption that more complex generative methods improve medical classifiers.
- Evidence and testing
- Ran systematic, controlled comparisons and chose methods on evidence rather than assumed superiority.
- Explaining failure
- Identified mode collapse and loss of diagnostic contrast, and backed conclusions with interpretability tools.
- Ownership
- The Academic Coordinator noted Sean's ownership, analytical depth and clear communication, and his support for peers.
Source: the Project Lead's and Academic Team's written feedback, 2025.
References
- Zhu, J.-Y. et al. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. ICCV.
- He, K. et al. (2016). Deep residual learning for image recognition. CVPR.
- Selvaraju, R. R. et al. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. ICCV.
- Lin, T.-Y. et al. (2017). Focal loss for dense object detection. ICCV.
Join the AI and Computer Vision in BioTech – Novo Nordisk Project
Work with a medical imaging scientist on a real question and leave with research you can defend.
