🎉 News • 📖 Overview • ✨ Getting Started • 📊 Data • 📌 Citation

• [2026-04-16] KnowRL ranks #1 on Hugging Face Daily Papers! Check it out: Daily Paper.
• [2026-04-15] We release our paper, code, training data, KP annotations, and model checkpoints. Check it out: Paper.

Hint-based reinforcement learning (RL) addresses reward sparsity in LLM reasoning by injecting auxiliary guidance into prompts. However, existing methods suffer from hint redundancy — they inject excessive or loosely structured guidance while only a small subset of information is actually needed. We identify three key challenges:

Figure 1. Three key challenges in hint-based RL. (a) The critical-segment effect: performance improves sharply once a short key hint segment appears, with diminishing returns beyond it. (b) Cross-hint inconsistency: longer prefixes may introduce branching or ambiguity. (c) Training-efficiency trade-off: abstraction-based hints rely on teacher models, increasing computational overhead.

KnowRL formulates hint design as a minimal sufficient guidance problem. Instead of injecting long solution prefixes or full reasoning templates, KnowRL decomposes guidance into atomic knowledge points (KPs) and identifies the minimal subset required to unlock reward learning.

1. Minimal-sufficiency perspective on hint-based RL — We empirically demonstrate a non-linear, jump-like performance pattern (critical-segment effect), revealing that effective guidance depends on selective key knowledge rather than cumulative hint length.

2. Principled KP selection pipelines — We design several KP selection strategies (S-LOO, T-LOO, CBRS, CSS) that ensure minimal, non-redundant, and interaction-compatible KP subsets. CSS achieves the best performance with ~38% fewer KPs.

3. State-of-the-art results at 1.5B scale — KnowRL-Nemotron-1.5B achieves 74.16 average accuracy with CSS across eight competition-level math benchmarks, establishing a new SOTA.

KnowRL-Nemotron-1.5B achieves consistent improvements across all eight benchmarks under different hint settings:

Even without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 (+9.63 over baseline), showing that KnowRL improves the underlying policy itself rather than relying on test-time hint injection.

We compare multiple offline KP selection strategies on Nemotron-1.5B. CSS achieves the highest accuracy with only 2.57 KPs per problem on average:

CSS-selected KPs deliver larger and more consistent gains across different difficulty levels compared to full-KP injection, which can even introduce negative effects on certain subsets:

Figure 2. Difficulty-bucket analysis on test set (left) and training set (right). CSS-selected KPs deliver larger and more consistent gains across difficulty levels. Full-KP injection can introduce regressions on certain subsets.

Figure 3. Comparison of KP selection strategies (CSS vs CBRS) under the same training budget. CSS maintains a persistent advantage in training accuracy with more stable policy refinement.

KnowRL training dramatically reduces reward sparsity: the zero-correct fraction drops from 41.21% to 13.00%, while the all-correct bucket rises from 1.35% to 34.28% (+32.93pp).

Figure 4. Distribution of per-query correct counts on the training set. KnowRL training (middle) collapses the zero-correct fraction and shifts mass toward all-correct. Adding KP hints at inference (right) further concentrates correctness.

We discover a pruning interaction paradox: removing individual "bad" KPs improves accuracy, but removing them jointly can degrade performance due to inter-KP dependencies.

Figure 5. Left: Pruning interaction paradox under LOO-style selection — cross-hint inconsistency occurs in 40%–60% of cases. Right: Tolerance-threshold sensitivity for the delta parameter in CBRS.

Figure 6. Visualization of the critical-segment effect across prefix ratios on 50 training instances. Performance typically remains flat in low-ratio regions, then exhibits a distinct jump once a key segment is included.

KnowRL/
├── README.md
├── setup_env.sh              # One-click environment setup script
├── environment.yml           # Conda environment configuration
├── requirements.txt          # Full pip dependencies for exact reproducibility
│
├── eval/                     # Evaluation pipeline
│   ├── data/                 # Evaluation benchmark datasets
│   │   ├── AIME24/           # AIME 2024
│   │   ├── AIME25/           # AIME 2025
│   │   ├── AMC23/            # AMC 2023
│   │   ├── BRUMO25/          # BRUMO 2025
│   │   ├── CMIMC25/          # CMIMC 2025
│   │   ├── HMMT25/           # HMMT 2025
│   │   ├── MATH_500/         # MATH-500 subset
│   │   └── Olympiad_Bench/   # Olympiad Bench
│   └── eval_scripts/         # Evaluation scripts
│       ├── task.py           # Task definitions (all supported task names)
│       ├── prompts.py        # Prompt templates (with/without KP hints)
│       ├── s1_gen_vllm.py    # Step 1: Generate responses via vLLM
│       ├── s1_gen_vllm.sh    # Step 1: Shell wrapper
│       ├── s2_vllm_serve.sh  # Step 2: Serve judge model (CompassVerifier-3B)
│       ├── rule_base_verl.py # Rule-based answer verification
│       ├── s3_rule_base_verl.sh  # Step 3: Run rule-based evaluation
│       ├── s3_model_base_verl.py # Model-based evaluation (with judge)
│       └── eval_outputs/     # Generated evaluation results
│
├── train/                    # Training pipeline
│   ├── knowrl.sh             # Main training launch script (DAPO/GRPO via Ray)
│   ├── train_data/           # Training data (KP hints embedded in prompts)
│   │   ├── css.jsonl
│   │   └── css.parquet
│   └── val/                  # Validation data used during training
│       ├── aime24/
│       ├── aime25/
│       ├── brumo_2025/
│       └── hmmt_25_2/
│
├── utils/
│   └── ray_start.sh          # Ray cluster initialization for multi-node training
│
├── huggingface/              # Scripts and data for HuggingFace upload
│   ├── hf_upload.py
│   └── train/
│
└── verl/                     # verl framework (RL training library for LLMs)

• Linux (tested on Ubuntu)
• NVIDIA GPU with CUDA 12.4+
• Conda (Miniconda or Anaconda)

git clone <knowrl_repo_url>
cd KnowRL

Note: This step only installs the Python dependencies. verl is installed separately in the next step.

bash setup_env.sh
conda activate knowrl

conda env create -f environment.yml
conda activate knowrl

Note: If pip fails to find torch, your pip mirror may not host PyTorch. The environment.yml includes --extra-index-url for the official PyTorch index, but if it still fails, use Option A or Option C instead.

# 1. Create conda env
conda create -n knowrl python=3.10 -y
conda activate knowrl

# 2. Install PyTorch (CUDA 12.4)
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
    --extra-index-url https://download.pytorch.org/whl/cu124

# 3. Install all other dependencies
pip install -r requirements.txt --extra-index-url https://pypi.org/simple/

# 4. Install flash-attn (requires compilation)
pip install flash-attn==2.7.4.post1 --no-build-isolation

cd verl
pip install -e . --extra-index-url https://pypi.org/simple/

python -c "import torch; print(f'PyTorch {torch.__version__}, CUDA {torch.cuda.is_available()}')"
python -c "import transformers; print(f'Transformers {transformers.__version__}')"
python -c "import vllm; print(f'vLLM {vllm.__version__}')"
python -c "import verl; print('verl OK')"

• eval/data/: Evaluation data for offline evaluation, consistent with the evaluation data on HuggingFace.

• train/train_data/: Training-ready data. Knowledge points are directly embedded in the prompt as a ## Hint section (rather than listed separately). Each record contains kp_list (all candidate knowledge points) and kept_kp_index (indices of selected knowledge points injected into the prompt). The HuggingFace version provides the complete knowledge points along with those extracted by CSS and CBRS strategies respectively.

• train/val/: Validation data used during training. Prompts contain only the math problem followed by the suffix Please reason step by step, and put your final answer within \boxed{}., without any knowledge point hints.

We use the DAPO/GRPO algorithm via verl with Ray for distributed training.

Base model: nvidia/OpenMath-Nemotron-1.5B

Key hyperparameters:

For single-node training, you can skip this step — Ray will be automatically initialized when the training script runs.

For multi-node training, start the Ray cluster before launching the training script on each node:

# On head node
bash utils/ray_start.sh

bash train/knowrl.sh

Training checkpoints will be saved under checkpoints/. Validation and rollout data will be saved under validation_data/ and rollout_data/ respectively. Training metrics are logged to WandB.

We apply entropy annealing by adjusting the clip upper bound during training. After 2,590 steps, clip_high is reduced from 0.28 to 0.26, inducing a faster entropy drop and encouraging the policy to shift from exploration to exploitation.

Figure 7. Comparison with and without entropy annealing. Entropy annealing yields faster entropy reduction and consistently better validation performance, contributing +0.74 average accuracy improvement.

The evaluation pipeline consists of three steps: generation, judge model serving, and scoring.

We evaluate on 8 benchmarks (1,374 problems total): AIME24, AIME25, BRUMO25, HMMT25, AMC23, CMIMC25, MATH-500, and Olympiad-Bench.

First, check all supported task names in eval/eval_scripts/task.py. The available tasks include:

• Raw (no KP hints): AIME24, AIME25, BRUMO25, HMMT25, AMC23, CMIMC25, MATH_500, Olympiad_Bench
• CBRS (Consensus-Based Robust Selection): AIME24_CBRS, AIME25_CBRS, ... (append _CBRS suffix)
• CSS (Constrained Subset Search): AIME24_CSS, AIME25_CSS, ... (append _CSS suffix)

Then modify eval/eval_scripts/s1_gen_vllm.sh to set your model path and task list, and run:

cd eval/eval_scripts
# Edit s1_gen_vllm.sh to configure your model and tasks
bash s1_gen_vllm.sh

Results will be saved under eval/eval_scripts/eval_outputs/.

Launch the CompassVerifier-3B as a judge model using vLLM:

export CUDA_VISIBLE_DEVICES=0
vllm serve <path_to_CompassVerifier-3B> \
    --served-model-name cv_3b \
    --tensor-parallel-size 1 \
    --pipeline-parallel-size 1 \
    --trust-remote-code \
    --port 8000 \
    --host 0.0.0.0

Run rule-based evaluation on the generated outputs:

bash eval/eval_scripts/s3_model_base_verl.sh

If you find this work helpful, please cite our paper:

@misc{yu2026knowrlboostingllmreasoning,
      title={KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance}, 
      author={Linhao Yu and Tianmeng Yang and Siyu Ding and Renren Jin and Naibin Gu and Xiangzhao Hao and Shuaiyi Nie and Deyi Xiong and Weichong Yin and Yu Sun and Hua Wu},
      year={2026},
      eprint={2604.12627},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2604.12627}, 
}