• Elyse Marie Uyiringiye
• Nice Eva Karabaranga
• Best Verie Iradukunda
• Raissa Irutingabo

[https://drive.google.com/drive/folders/18KEPQ7AlXoZxDiDu8lFAwod63G7g-8rQ?usp=sharing]

Best Model play

This project implements a Deep Q-Network (DQN) agent trained to play the Atari Boxing-v5 environment using Stable Baselines3 and Gymnasium. This requires training and comparing multiple DQN configurations with different hyperparameters and policy architectures (CNNPolicy vs MLPPolicy).

• Gymnasium Atari environment: ALE/Boxing-v5
• Framework: Stable-Baselines3 DQN

• scripts/train.py: trains different DQN experiments and saves model artifacts
• scripts/play.py: loads trained best model and runs evaluation gameplay

stable-baselines3 (≥2.3.2): DQN algorithm and policy implementations gymnasium[atari] (≥1.0.0): Atari environment wrapper ale-py (≥0.10.1): Atari Learning Environment interface torch: Neural network backend pandas, matplotlib: Data analysis and visualization

pip install -r requirements.txt

AutoROM --accept-license

The best-performing model is:boxing_exp01_baseline_cnn with mean reward of 5.9

The baseline CNN model outperformed others because it used balanced default hyperparameters that ensured stable learning. With only 100,000 timesteps, modified settings like high gamma or altered exploration failed to converge. The CNN also captured spatial features effectively, making it more robust than other configurations, especially under limited training conditions.

• experiments/BestVerie_experiments.ipynb: Full training pipeline with callbacks and visualization
• Hyperparameter_tables/BestVerie_hyperparameter_results.csv: Summary results table
• In drive link: /BestVerie.zip: logs,models, everything related to the training process

Raissa Experiments (10 Configurations)Notebook used: experiments/irutingabo-experiments.ipynbThis section defines 10 experiment configurations arranged as 5 paired CNN/MLP groups — each pair shares identical hyperparameters so that CnnPolicy and MlpPolicy can be compared fairly under the same conditions.CSV Results: results/raissa/tables/raissa_hyperparameter_results.csvMarkdown Results: results/raissa/tables/raissa_hyperparameter_results.md

Findings and Insights

Best model: Raissa Exp10 MLP Large Batch, with a mean reward of 2.8. It was the only experiment to achieve a positive score across all 10 runs.

Surprising result: MLP outperformed CNN overall. The average mean reward across all five CNN experiments was -6.04, while the MLP average was -5.68. This is unexpected since Boxing is a pixel-based environment. However, with only 50,000 timesteps, the CNN likely did not have enough time to learn useful visual features. In contrast, the MLP trained faster (about 1.5 minutes vs 5 minutes) and made better use of the limited training budget.

What helped: Lower learning rate. Reducing the learning rate to 5e-5 produced the best CNN result (-1.2) and the second-best MLP result (-2.4). More cautious updates appear to stabilize training under a short timestep budget.

What helped: Larger batch size for MLP. The MLP with a batch size of 128 performed best overall. Smoother gradient estimates provided a stronger learning signal, allowing the model to develop a basic positive strategy.

What hurt: Higher learning rate. At 2.5e-4, updates were too aggressive, leading to unstable training and inconsistent Q-value estimates.

What hurt the most: Slow epsilon decay. Extending exploration to 50% of training was the worst-performing decision. Spending too much of a short 50k-step budget on random exploration left insufficient time for effective exploitation.

The best-performing model was exp09_fast_eps (CnnPolicy) with a mean reward of 11.2, being the highest in the group. Fast epsilon decay worked well here because Boxing has a simple and dense reward signal; the agent receives feedback on every punch, so it learns a good strategy quickly and benefits more from exploiting that strategy early rather than spending time on random exploration. The high learning rate experiment (lr=5e-4) was the worst, scoring -41.0, because overly large updates destabilised the Q-values and the agent never converged. The CNN policy consistently outperformed MLP across all comparable experiments with identical settings. CNN scored 3.8 while MLP scored -33.2, confirming that convolutional layers are essential for pixel-based games where spatial understanding of the opponent's position and movement is critical to learning an effective policy.

Watch the agent play Boxing

Agent uses GreedyQPolicy (exploration_rate=0.0 → always picks argmax Q(s,a)).

• Best config: exp02_zero_eps_end :setting epsilon_end to 0.0 forces fully greedy exploitation at the end of training, giving the highest reward
• Worst config: exp04_very_high_lr : lr=1e-3 caused Q-value instability, scoring -24.80
• Zero gamma finding: Setting gamma=0.0 makes the agent fully myopic, meaning it only optimises for immediate reward and cannot learn long-term boxing strategy
• Exploration tradeoff: Both instant exploitation (exp07) and full exploration (exp08) underperformed, confirming that a balanced epsilon decay is important

• Notebook: experiments/Nice_experiments.ipynb
• Best model: Nice_dqn_model.zip
• Hyperparameter table: hyperparameter_table_nice.csv
• Reward comparison: assets/nice_reward_comparison.png
• Training curves: assets/nice_training_curves.png

My best model