A lightweight post-training framework for LLMs and VLMs. Maximizing developer speed. Scales to billions of parameters with DeepSpeed, vLLM, and Ray.

Context-length minimized principle. Following LLM Oriented Design. So your LLM agent will not suffer from OOT or IQ loss problem.

Post-train any model with one command. oxRL auto-detects your hardware, auto-prepares datasets, and scales to multi-GPU automatically.

# CLI — one command to train
oxrl train --model Qwen/Qwen2.5-0.5B-Instruct --task math --epochs 3

# With a custom dataset (JSONL or Parquet)
oxrl train --model Qwen/Qwen2.5-7B-Instruct --dataset ./my_data.jsonl --task reasoning

# Python API — under 3 lines
from oxrl import Trainer

trainer = Trainer(model="deepseek-ai/DeepSeek-R1-Distill-Llama-8B")
trainer.train(task="reasoning")

Datasets can be Parquet or JSONL — format is auto-detected from the file extension.

oxRL works with any HuggingFace model that supports AutoModelForCausalLM, including multimodal models via AutoModelForImageTextToText. No special integration needed — just pass the model name.

These models have been explicitly verified through our automated onboarding pipeline:

┌──────────────────────────────────────────────────────────────────┐
│                          oxRL Framework                          │
├────────────────────────────────┬─────────────────────────────────┤
│     RL Path (main_rl.py)       │     SL Path (main_sl.py)        │
│  SGRPO / GSPO / CISPO / PPO   │  SFT / DPO / ORPO / KTO         │
│  RLHF / RLAIF                  │  CPT / KD / RM / RFT            │
│  Ray actors + vLLM rollouts    │  OnlineDPO / SPIN / IPO / SimPO │
│                                │  DeepSpeed distributed training │
├────────────────────────────────┴─────────────────────────────────┤
│  oxrl/algs/       Algorithms   │  oxrl/rollouts/   vLLM + Replay │
│  oxrl/configs/    Pydantic cfg │  oxrl/rewards/    Verifiable    │
│  oxrl/datasets/   HF loaders   │  oxrl/utils/      Setup + Logs  │
└──────────────────────────────────────────────────────────────────┘

1. Scout Agent: Discovers model metadata and ensures chat_template compatibility.
2. Multimodal Pipeline: Converts base64 images/audio into PIL/NumPy for vLLM rollouts.
3. LoRA Lifecycle: Train with adapters, save with gathered ZeRO-3 weights, and auto-strip PEFT prefixes for immediate vLLM compatibility.
4. Verifiable Rewards: Programmatic verification of CoT tags and mathematical correctness.

# From source (recommended for development)
git clone https://github.com/warlockee/oxRL.git
cd oxRL
pip install -e .

# Or from PyPI
pip install oxrl

pip install pytest
pytest tests/test_bugs.py -v

Before starting a long training run, verify your environment (GPUs, CUDA Toolkit, DeepSpeed, Ray) with our diagnostic tool:

oxrl doctor

oxRL uses YAML config files. See oxrl/configs/rl_args.yaml (RL) and oxrl/configs/sl_args.yaml (SL) for all available options with documentation. Example configs are in registry/examples/.

Key environment variables:

• OXRL_DATA_DIR — Override default data directory (default: ./data)
• OXRL_CHECKPOINT_DIR — Override default checkpoint directory (default: ./checkpoints)
• HF_TOKEN — HuggingFace token for gated models
• GITHUB_TOKEN — For autonomous bug reporting (optional)

# config.yaml
model:
  name: "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
lora:
  enabled: true
reward:
  reward_func: "reasoning_reward_func"
data:
  dataset: "openr1_math"

python main_rl.py --config-file config.yaml

All reward functions share the signature (prompt_ids, response_ids, finish_reason, metadata) → (rewards, is_per_token). Set via reward_func in your config YAML.

oxRL/
├── oxrl/                   # Core Framework Package
│   ├── trainer.py          # High-level Trainer API
│   ├── rewards/            # Verifiable reasoning and coding rewards (math, code, etc.)
│   ├── algs/               # 51 algorithm implementations (see tables above)
│   ├── swarm/              # Autonomous model onboarding (Scout, Bugfixer)
│   ├── preprocessing/      # Reasoning (OpenR1), Multimodal (Vision/Audio) preprocessors
│   ├── rollouts/           # vLLM inference with structured prompt support
│   └── datasets/           # Dataset loaders and samplers
├── main_rl.py              # RL training loop (Ray + DeepSpeed)
├── main_sl.py              # SL training loop (DeepSpeed) — 12 algorithms
├── registry/examples/      # Example configs for all 18 algorithms
├── examples/               # Ready-to-use recipes and training scripts
└── pyproject.toml          # Packaging and Installation

Debuggability over Pipelining. oxRL avoids complex async pipelining to ensure that failure states are 100% reproducible and logs are clear.

Robust Environment Handling. oxRL is designed to work even in constrained environments. It automatically handles common CUDA/DeepSpeed mismatches by providing actionable warnings instead of fatal crashes.

Autonomous Bug Reporting. On framework failure, oxRL provides structured diagnostic signals for AI agents to automatically generate and submit GitHub issues (requires GITHUB_TOKEN environment variable).

LoRA-first for 7B+. We default to LoRA for larger models to enable high-quality research on consumer-grade and restricted high-end hardware.

Verification-driven RL. We prioritize datasets where the reward is verifiable (Math, Code, Format) to drive logical discovery.

This repository is optimized for LLM-assisted development (Claude/Gemini). If you are asking an AI to work on this framework, refer them to these "High-Signal" files:

• Bug Reporting: See BUG_REPORTING.md for instructions on autonomous issue submission.
• Adding a New Algorithm: See oxrl/algs/base.py (Base Class) and oxrl/algs/grpo.py (Implementation).
• Adding a Reward Function: Add to oxrl/rewards/ using the signature in oxrl/rewards/base.py.
• Changing Model Loading: See oxrl/utils/setup.py -> load_model_and_ref.
• Training Logic: RL loop in main_rl.py, SL loop in main_sl.py.
• Config Validation: Logic is in oxrl/configs/load.py.

Contributions are welcome. Please follow the existing architectural patterns and style.

Check out the FAQ for details on LoRA merging and Multimodal input formatting.

There is a growing claim in the community that the concept of a "rollout" is outdated and that inference servers are the new paradigm. We disagree. Here is our analysis of the debate and why oxRL is designed the way it is.

"Rollout is outdated. Inference server is the new trend."

An inference server is not a replacement for rollouts — it is an implementation strategy for rollouts. Every RL training loop for LLMs still performs rollouts: the policy model autoregressively generates trajectories that are scored and used for gradient updates. The question is only where and how that generation happens.

Both approaches still generate rollouts. The "inference server" camp is running rollouts on a dedicated vLLM/SGLang process instead of in-process. That is an infrastructure decision, not a conceptual paradigm shift.

1. NVIDIA is scaling rollouts up, not phasing them out. BroRL pushes rollout counts from N=16 to N=512 per prompt and demonstrates this breaks through performance ceilings where step-scaling cannot. If rollouts were obsolete, the latest NVIDIA research would not make them the primary scaling axis.

2. Disaggregation has real costs. Serialization overhead, network bandwidth for weight sync, and added orchestration complexity. Co-located approaches like veRL's HybridEngine avoid weight transfer overhead entirely by keeping generation and training on the same devices.

3. The motivation for disaggregation is agentic RL, not rollout obsolescence. Co-located rollouts break down when you need multi-turn tool use, because synchronous batch processing cannot handle variable-latency tool calls. That is a real constraint for agentic workloads — but it is a specific use case, not a universal indictment of rollouts.

4. "Inference server" is rebranding. Putting an HTTP endpoint in front of your rollout worker does not change the fundamental computation. StreamRL achieves a 2.66x throughput improvement through stream generation and skewness-aware scheduling — engineering wins on the rollout pipeline, not alternatives to it.

oxRL uses co-located vLLM rollouts by design. For standard RL post-training (GRPO, PPO, RLHF) on math, reasoning, and coding tasks — which is the vast majority of practical post-training — this architecture is simpler, avoids network overhead, and produces fully reproducible training runs. We prioritize debuggability over pipelining.

If and when agentic multi-turn RL with tool use becomes the dominant training paradigm, disaggregated inference will be the right call. Until then, saying "rollout is outdated" is like saying "compilation is outdated because we use build servers now." The build server does the compilation. The inference server does the rollout. The abstraction changed; the computation did not.

If you find oxRL useful in your research, please cite our paper:

@article{oxrl2026,
  title={Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales with Scale-Dependent Ranking Inversions},
  author={Li, Xiaoyi},
  journal={arXiv preprint arXiv:2603.19335},
  year={2026},
  url={https://arxiv.org/abs/2603.19335}
}