Course: Data Mining | FAST NUCES
Dataset: Kaggle — Chest X-Ray Images (Pneumonia)
Model: MobileNetV2 Transfer Learning
Hugging Face: pneumonia-detection-model

DEMO:

The project runs on WSL2 (Ubuntu) with GPU support.
TensorFlow 2.11+ dropped native Windows GPU — WSL2 is required.

# Run in PowerShell as Administrator
wsl --install

Restart your machine. To open WSL:

wsl -d Ubuntu

The venv must be created in the WSL home directory. Creating on /mnt/c/ fails due to Windows filesystem permission restrictions.

sudo apt update && sudo apt install -y python3 python3-pip python3-venv

cd ~
python3 -m venv xray_venv

source ~/xray_venv/bin/activate

Run each session after activating:

export LD_LIBRARY_PATH=$(find ~/xray_venv -path "*/nvidia/*/lib" -type d 2>/dev/null | tr '\n' ':')$LD_LIBRARY_PATH

Make permanent:

echo 'export LD_LIBRARY_PATH=$(find ~/xray_venv -path "*/nvidia/*/lib" -type d 2>/dev/null | tr '"'"'\n'"'"' '"'"':'"'"')$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Verify:

python3 -c "import tensorflow as tf; print('GPUs:', tf.config.list_physical_devices('GPU'))"
# Expected: GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

source ~/xray_venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

requirements.txt

tensorflow[and-cuda]
opencv-python
numpy
matplotlib
scikit-image
Pillow
seaborn
tqdm
scikit-learn

Source: Guangzhou Women and Children's Medical Center, via Kaggle.
All images are anterior-posterior chest X-rays in JPEG format.

Class Imbalance Handling

Data augmentation was trialled but caused NORMAL recall to drop to 0.39. 65% of NORMAL training images became synthetic, and the model learned augmentation artefacts instead of real anatomy. Augmentation was abandoned. Class imbalance is handled entirely through compute_class_weight('balanced') in model.fit(), which assigns NORMAL a weight of ~1.95 and PNEUMONIA ~0.67.

The same five-step pipeline is applied identically to train, validation, and test images.

Original  →  Bicubic Resize (128×128)  →  Median Blur (k=3)
          →  CLAHE (clipLimit=2.0)     →  Z-Score Normalise  →  Save as .npy

This was the most critical decision. Saving as uint8 PNG silently undoes z-score normalisation:

# What PNG save does internally:
img = ((img - img.min()) / (img.max() - img.min()) * 255).astype(np.uint8)
# This is min-max rescaling — restores absolute brightness per image

The result was a domain shift of 26.7 intensity units between train NORMAL (mean 124.9) and test NORMAL (mean 151.6). The model learned brightness as a class proxy. Test NORMAL images from a different scanner were brighter and misclassified as PNEUMONIA.

Saving as float32 .npy preserves exact values. After the fix, the domain shift dropped to < 0.1 units.

def preprocess_image(img_path, target_size=(128, 128)):
    img = cv2.imread(str(img_path), cv2.IMREAD_GRAYSCALE)
    img = cv2.resize(img, target_size, interpolation=cv2.INTER_CUBIC)
    img = cv2.medianBlur(img, 3)
    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
    img = clahe.apply(img)
    img = img.astype(np.float32)
    img = (img - np.mean(img)) / (np.std(img) + 1e-7)
    return img

def save_as_npy(img_array, save_path):
    np.save(str(save_path), img_array.astype(np.float32))

Just run pre-processing.ipynb file

Twenty-three techniques were evaluated. Each was assessed using visual quality and histogram fidelity. A smooth continuous histogram indicates preserved diagnostic information. Spikes and gaps indicate destroyed information.

Early-stage pneumonia may appear as only a 3–5 intensity unit difference from healthy tissue. Any technique that clusters or merges nearby intensity values poses a direct diagnostic risk.

Both methods were rejected. Pneumonia is diffuse opacity — sharpening amplifies noise to the same degree as real edges and can create false anatomical boundaries.

All three rejected. Edge detection converts the image to a binary map and discards all soft-tissue information. Pneumonia presents as diffuse cloud-like opacity — the exact information that edge detection removes.

MobileNetV2 Transfer Learning was chosen over a custom CNN.

A custom CNN trained from scratch on ~5,000 images performed poorly (NORMAL recall 0.52 at best). MobileNetV2 was pretrained on 1.4 million ImageNet images and already understands edges, textures, and structural patterns. Fine-tuning it for chest X-ray classification is substantially more effective.

Phase 1 — Head only (base frozen)

MobileNetV2 base fully frozen. Only Dense head trains. Learns basic NORMAL vs PNEUMONIA separation without touching pretrained weights.

Epochs:        20  (early stopping patience=5)
Learning rate: 1e-4
Monitor:       val_auc

Phase 2 — Fine-tuning (last 14 layers)

Layers 140–154 of MobileNetV2 unfrozen. These learn chest X-ray specific features. Very low LR to avoid destroying ImageNet knowledge. Previous attempts with 54 unfrozen layers caused train loss 0.023 vs test loss 2.8 (severe overfitting). Limiting to 14 layers resolved this.

Epochs:          10
Learning rate:   5e-6
Unfrozen layers: 14 of 154

source ~/xray_venv/bin/activate
export LD_LIBRARY_PATH=$(find ~/xray_venv -path "*/nvidia/*/lib" -type d 2>/dev/null | tr '\n' ':')$LD_LIBRARY_PATH
cd /mnt/c/Users/ahmad/Downloads/quiz/DM-Pneumonia-Detection
python3 model/train.py

The green dashed line marks where Phase 2 fine-tuning begins.

Phase 1 — Epoch log (selected)

Phase 2 — Fine-tune epoch log (selected)

training_logs/
├── training_history_phase1.csv
├── training_history_phase2.csv
├── training_curves.png
├── confusion_matrix.png
└── sample_predictions.png