A production-grade Deep Reinforcement Learning (DRL) framework for stock trading on the Brazilian stock exchange (B3) with Physics-Informed Neural Networks (PINN) integration and ensemble learning strategy.

This project implements a sophisticated ensemble of DRL agents (PPO, DDPG, A2C) that learn to trade stocks through end-to-end reinforcement learning. The system is designed for research and production use, featuring:

• ✅ Multi-agent ensemble with weighted voting based on historical performance
• ✅ Rolling window cross-validation for robust out-of-sample evaluation
• ✅ Bayesian hyperparameter optimization via Optuna (TPE sampler)
• ✅ PINN integration for extracting market-driven features (optional)
• ✅ Comprehensive evaluation with Pyfolio tear sheets
• ✅ Timeout protection with automatic checkpointing for stability
• ✅ Professional visualizations (returns, drawdown, metrics, tear sheets)
• ✅ Reproducible results with deterministic seed management

Key Innovation: The ensemble voting strategy combines PPO (stability), DDPG (off-policy efficiency), and A2C (fast learning) to reduce overfitting and improve robustness across market regimes.

DRL_ICproject/
├── config/
│   ├── config.py                    # Global paths, assets, capital
│   └── hyperparameters.py           # Algorithm parameters + training config
├── src/
│   ├── agents/
│   │   ├── base_agent.py            # Abstract base class
│   │   ├── drl_agents.py            # PPO, DDPG, A2C implementations
│   │   ├── ensemble_agent.py        # Ensemble voting strategy
│   │   ├── training_utils.py        # Timeout handler + checkpointing
│   │   └── models.py                # Custom NN architectures
│   ├── data/
│   │   ├── data_loader.py           # CSV loading & validation
│   │   ├── data_processor.py        # Technical indicator calculation
│   │   └── rolling_window_strategy.py    # Cross-validation windows
│   ├── env/
│   │   └── stock_trading_env.py     # Gymnasium-compatible environment
│   ├── evaluation/
│   │   ├── results_manager.py       # Save/load metrics & models
│   │   └── visualization.py         # Matplotlib + Pyfolio integration
│   ├── optimization/
│   │   └── hyperparameter_optimizer.py  # Optuna Bayesian search
│   ├── core/
│   │   ├── reproducibility.py       # Deterministic seed management
│   │   └── validation.py            # Safety checks
│   └── pinn/
│       ├── inference_wrapper.py     # PINN feature extraction
│       └── ...
├── data/
│   ├── raw/                         # CSV files from B3
│   ├── processed/                   # Preprocessed cache
│   └── pinn_features/               # Heston parameters
├── trained_models/                  # Saved agent checkpoints
├── results/
│   ├── metrics/                     # JSON/CSV metrics (timestamped)
│   ├── models/                      # Final trained models
│   └── plots/                       # Generated visualizations
├── examples/                        # Example scripts
├── tests/                           # Unit tests
├── main.py                          # Master orchestration script
├── PIPELINE_EXECUTION_GUIDE.md      # Detailed execution flow
├── TRAINING_ENHANCEMENTS.md         # Timeout & optimization guide
├── RESULTS_SAVING_GUIDE.md          # Results persistence guide
├── ARCHITECTURE.md                  # Design documentation
├── requirements.txt
└── README.md                        # This file

# Clone repository
git clone https://github.com/your-org/drl-trading-agent.git
cd drl-trading-agent

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Place CSV files in data/raw/ with format: {ASSET}.csv

Expected columns:

timestamp,open,high,low,close,volume
2023-01-03 10:00:00,28.50,28.75,28.40,28.60,1500000
2023-01-03 10:01:00,28.61,28.80,28.55,28.70,1200000

Optional columns:

• dividend_yield: For dividend-adjusted analysis
• risk_free_rate: For options Greeks calculation

# Default: Rolling window ensemble (18 windows, ~2-3 min)
python main.py

# Simple 80/20 split (~60 sec)
python main.py --mode simple-pipeline --assets PETR4

# Bayesian hyperparameter optimization (~10-20 min)
python main.py --mode optuna-optimize --agent-type PPO --n-trials 20

# View all options
python main.py --help

When: Rapid prototyping, hyperparameter tuning
Duration: ~60 seconds on CPU
Strategy: 80/20 train/test split

python main.py --mode simple-pipeline --assets PETR4 VALE3

Flow:

Load Data → Feature Engineering → Create Agents (PPO, DDPG, A2C)
→ Train on 80% → Evaluate on 20% → Ensemble → Save Results

When: Robust out-of-sample evaluation, production use
Duration: ~2-3 minutes with 18 windows
Strategy: 18 sliding windows (14-week train, 4-week test)

python main.py --mode rolling-ensemble --assets PETR4 VALE3

Flow:

For each of 18 windows:
  → Create Train/Test split (14/4 weeks, 7-week overlap)
  → Train 3 Agents → Evaluate → Create Ensemble
  → Record metrics (Sharpe, Drawdown, Win Rate, etc.)
→ Aggregate statistics across windows

Window Specification:

• Training: 98 trading days (14 weeks)
• Testing: 28 trading days (4 weeks)
• Stride: 7 days (7-week overlap with previous)
• Ideal for: 2-3 years of 1-minute OHLCV data

When: Finding best hyperparameters, research
Duration: ~10-20 minutes (20 trials)
Strategy: Optuna TPE sampler with MedianPruner

python main.py --mode optuna-optimize --agent-type PPO --n-trials 30

Search Space (per algorithm):

Sampler: Tree Parzen Estimator (Bayesian optimization)
Pruner: Median Pruner (early stopping for unpromising trials)

• Best for: Stability, sample efficiency, variance reduction
• Action space: Continuous [-1, 1] allocation per asset
• Key innovation: Clipping objective prevents large policy updates
• Reference: Schulman et al., 2017

• Best for: Off-policy learning, sample efficiency
• Action space: Continuous [-1, 1] allocation per asset
• Key innovation: Target network + replay buffer for stability
• Reference: Lillicrap et al., 2016

• Best for: Fast learning, parallel data collection
• Action space: Continuous [-1, 1] allocation per asset
• Key innovation: Synchronous parallel environments
• Reference: Mnih et al., 2016

• Voting mechanism: Weighted average of agent actions
• Weights: Normalized by mean_reward from validation
• Rationale: Reduce overfitting, increase robustness across regimes

Total observation size: 44 dimensions

Composition:

• 5 price features: open, high, low, close, volume
• 7 technical indicators (14 values with MA windows):
  • SMA (5, 10, 20-day)
  • RSI (14-day momentum)
  • MACD (trend)
  • Bollinger Bands (volatility)
  • VWAP + ATR (volume/volatility)
  • Log Returns (daily %)
• 12 portfolio state features: cash, position_value, realized_pnl, etc.
• 6 aggregate indicators: min_price, max_price, price_range, etc.

Normalization:

• Zero-mean, unit-variance (StandardScaler)
• Clipped to [-3, 3] to handle outliers

All results saved with automatic timestamps to results/:

results/
├── metrics/
│   ├── simple_pipeline_metrics_20260209_192000.json
│   ├── rolling_ensemble_metrics_20260209_193000.csv
│   └── optuna_PPO_20260209_194000.json
├── models/
│   ├── ppo/
│   │   ├── ppo_20260209_192030/
│   │   └── metadata.json
│   ├── ddpg/ddpg_20260209_192030/
│   └── a2c/a2c_20260209_192030/
└── plots/
    ├── returns_distribution_20260209.png
    ├── drawdown_underwater_20260209.png
    └── metrics_comparison_20260209.png

{
  "mean_reward": 0.00342,
  "std_reward": 0.01565,
  "sharpe_ratio": 1.24,
  "sortino_ratio": 1.82,
  "calmar_ratio": 3.15,
  "max_drawdown": -0.125,
  "cumulative_return": 0.285,
  "win_rate": 0.583,
  "best_day": 0.0358,
  "worst_day": -0.0246,
  "trades_executed": 145,
  "avg_trade_pnl": 0.00196
}

from src.evaluation.results_manager import ResultsManager

mgr = ResultsManager(Path("results"))

# Load metrics
metrics = mgr.load_metrics(Path("results/metrics/simple_pipeline_metrics.json"))

# List available results
all_results = mgr.list_results()
print(all_results['models'])

# Get latest model
latest = mgr.get_latest_model(agent_name="ppo")

from src.evaluation.visualization import TradingVisualizer

vis = TradingVisualizer()

# 1. Portfolio value vs benchmark
fig = vis.plot_portfolio_value(results_df, benchmark_df)

# 2. Returns distribution + CDF
fig = vis.plot_returns_distribution(returns)

# 3. Drawdown underwater plot
fig = vis.plot_drawdown(returns)

# 4. Buy/sell signals on price chart
fig = vis.plot_actions(prices_df, actions_df)

# 5. Multi-agent metrics comparison
fig = vis.plot_metrics_comparison(metrics_dict)

# 6. Pyfolio tear sheet
metrics = vis.generate_pyfolio_report(returns, benchmark_returns)

# 7. Custom 2x2 performance grid
fig = vis.plot_performance_grid(results_dict)

Includes:

• Cumulative returns over time
• Rolling Sharpe ratio
• Underwater plot (drawdown timeline)
• Daily returns distribution
• Monthly returns heatmap
• Best/worst days
• Win rate and Calmar ratio

# Portfolio settings
PRIMARY_ASSETS = ['PETR4', 'VALE3', 'ABEV3']
INITIAL_CAPITAL = 100_000          # R$ 100,000
TRANSACTION_COST = 0.0005          # 0.05% (B3 typical)
SLIPPAGE = 0.001                   # 0.1%

# Rolling window specification
ROLLING_WINDOW_CONFIG = {
    'train_days': 98,              # 14 weeks
    'test_days': 28,               # 4 weeks
    'step_days': 7,                # 1 week overlap (stride)
}

# Data paths
DATA_PATH = Path(__file__).parent.parent / "data"
MODELS_PATH = Path(__file__).parent.parent / "trained_models"

# PPO agent
PPO_PARAMS = {
    'learning_rate': 3e-4,
    'n_steps': 2048,
    'batch_size': 64,
    'n_epochs': 10,
    'gamma': 0.99,
    'gae_lambda': 0.95,
    'clip_range': 0.2,
}

# DDPG agent
DDPG_PARAMS = {
    'learning_rate': 1e-3,
    'batch_size': 256,
    'tau': 0.005,
    'gamma': 0.99,
    'action_noise': 0.1,
}

# A2C agent
A2C_PARAMS = {
    'learning_rate': 7e-4,
    'n_steps': 5,
    'gamma': 0.99,
    'gae_lambda': 0.95,
}

# Training configuration
TRAINING_CONFIG = {
    'timeout_seconds': 600,        # 10-minute timeout for DDPG
    'checkpoint_interval': 10_000,  # Save every 10k steps
    'keep_checkpoints': 3,          # Keep 3 most recent
}

Problem: DDPG can hang during experience replay updates
Solution: Automatic timeout with graceful fallback

from src.agents.training_utils import safe_train_with_timeout

success = safe_train_with_timeout(
    model=agent.model,
    total_timesteps=50_000,
    timeout_seconds=600,  # 10 minutes
    callback=lambda: checkpoint_manager.save(model, step)
)

if not success:
    logger.warning("Training timeout - using partially trained model")
    # Agent continues with current weights

Features:

• Cross-platform (threading-based, no signals)
• Graceful execution: saves before interrupt
• Configurable per algorithm

All runs are fully deterministic:

from src.core.reproducibility import set_all_seeds

set_all_seeds(seed=42)
# Sets: Python, NumPy, PyTorch, CUDA, Gymnasium seeds

Guarantee: Same data + same seed = identical results
Use case: Reproduce papers, compare algorithm variations

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_agents.py -k "test_ppo" -v

# Run with coverage
pytest tests/ --cov=src --cov-report=html

# Quick demonstration
python examples/00_quickstart.py

# Full pipeline
python examples/10_complete_pipeline_with_saving.py

# Data exploration
python examples/01_data_pipeline.py

Core ML Stack:

• stable-baselines3[extra] - State-of-art RL algorithms
• gymnasium - Gym environment standard
• torch - Deep learning backend (optional GPU)

Data & Numerics:

• pandas - Data manipulation
• numpy - Numerical operations
• pandas_ta - Technical analysis indicators

Optimization & Visualization:

• optuna - Hyperparameter optimization
• matplotlib - Plotting
• pyfolio - Trading tear sheets

See requirements.txt for full list with versions.

• PIPELINE_EXECUTION_GUIDE.md - Detailed 18-step execution flow with timing
• TRAINING_ENHANCEMENTS.md - Timeout, checkpointing, Optuna integration
• RESULTS_SAVING_GUIDE.md - Results persistence and visualization
• ARCHITECTURE.md - System design and component interactions
• Docstrings - Inline documentation in all modules

Enable Physics-Informed Neural Networks for market-driven features:

python main.py --pinn-features --assets PETR4 VALE3

Learns: Heston stochastic volatility parameters

• ν (instantaneous variance)
• θ (long-term variance)
• κ (mean reversion speed)
• ξ (volatility of volatility)
• ρ (spot-volatility correlation)

See src/pinn/ for implementation.

Benchmark (PETR4, 2-year period):

Observations:

• Ensemble captures strengths of each agent
• Consistent outperformance vs buy-and-hold
• Performance varies by market regime
• Volatility clustering periods more challenging

How to contribute:

1. Fork repository
2. Create feature branch: git checkout -b feature/your-feature
3. Implement changes with tests
4. Follow PEP 8 (use black formatter)
5. Add documentation
6. Submit pull request

Guidelines:

• Write unit tests for new features
• Update docstrings
• Run pytest before PR
• Reference relevant issues

MIT License - Free for academic and commercial use
See LICENSE for details

• Issues: GitHub Issues tab
• Discussions: GitHub Discussions
• Email: [email protected]

@software{drl_trading_2026,
  author = {Your Institution},
  title = {DRL Stock Trading Agent with PINN Integration},
  year = {2026},
  url = {https://github.com/your-org/drl-trading-agent}
}

Core Algorithms:

• Schulman et al. (2017) - PPO
• Lillicrap et al. (2016) - DDPG
• Mnih et al. (2016) - A3C/A2C

Physics-Informed ML:

• Raissi et al. (2019) - PINNs
• Heston (1993) - Stochastic volatility model

Trading & Finance:

• Avellaneda & Lee (2010) - High-frequency trading microstructure
• Sharpe (1966) - Risk-adjusted performance metrics

Libraries:

• Stable-Baselines3 docs
• Gymnasium docs
• Optuna docs

• Stable-Baselines3 team for excellent RL implementations
• Quantopian community for Pyfolio and trading insights
• B3 (Brazilian Securities Exchange) for market data
• Contributors and community for feedback

Status: Active Development
Version: 2.1.0
Last Updated: February 9, 2026
Python: 3.9+
License: MIT