A novel retrieval-based pre-training approach to bridge the gap between pre-trained language models and machine reading comprehension by strengthening evidence extraction capabilities.

Fangkai Jiao¹, Yangyang Guo¹, Yilin Niu², Feng Ji³, Feng-Lin Li³, Liqiang Nie^1*

¹ School of Computer Science and Technology, Shandong University, Qingdao, China
² Department of Computer Science and Technology, Tsinghua University, Beijing, China
³ Damo Academy, Alibaba Group, Hangzhou, China
* Corresponding author

• Paper: ACL/IJCNLP 2021 Findings | arXiv
• Code Repository: GitHub
• Checkpoints: Google Drive

• Updates
• Introduction
• Highlights
• Method / Framework
• Project Structure
• Installation
• Checkpoints / Models
• Dataset / Benchmark
• Usage
• Experiments Scripts and Configs
• Citation
• Acknowledgement
• License

• [05/2021] Initial release of the paper on arXiv.
• [05/2021] Release of code and pre-trained checkpoints (BERT and RoBERTa).

This is the official implementation of the paper REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training.

Pre-trained Language Models (PLMs) often struggle with evidence extraction, which requires reasoning across multiple sentences in Machine Reading Comprehension (MRC). To address this, we propose REPT, a framework that introduces two self-supervised tasks: Surrounding Sentences Prediction (SSP) and Retrieval-Based Masked Language Modeling (RMLM). These tasks allow PLMs to learn evidence extraction during pre-training, which is then seamlessly inherited by downstream MRC tasks.

• Effective Evidence Extraction: Specifically designed to enhance reasoning across multiple sentences without explicit human supervision.
• Cross-Architecture Compatibility: Successfully applied to both BERT and RoBERTa backbones.
• Robust Performance: Outperforms standard PLMs on five major MRC benchmarks including RACE, DREAM, Multi-RC, Hotpot QA, and SQuAD 2.0.

The REPT framework utilizes a pre-trained Transformer encoder and a multi-head attention-based query generator. It employs sentence-level retrieval for the SSP task and document-level retrieval for the RMLM task, ensuring the model learns to identify and utilize relevant context for answer prediction.

.
├── configs/               # Configuration files for pre-training and fine-tuning
├── processors/            # Data processing and Wikipedia pre-processing scripts
├── scripts/               # Bash scripts for task-specific fine-tuning
├── main_wiki_pretrain.py  # Standard pre-training entry point
└── ...

git clone https://github.com/iLearn-Lab/ACL21-REPT
cd ACL21-REPT

The code requires PyTorch. We recommend the following versions used in our experiments:

• Pre-training: torch==1.6.0 (or torch==1.5.1 for specific distributed environments).
• Dependencies: transformers, numpy, scipy.

Pre-trained BERT and RoBERTa models are available for download:

• Download Link: Google Drive Checkpoints

Place the downloaded checkpoints in your local directory and update the model_name_or_path in the relevant configs/ files.

We use the English Wikipedia dump. To prepare the data:

1. Download the Wikipedia dump.
2. Run the pre-processing script:

cd processors
python wiki_en_pretrain_processor.py

The model is evaluated on:

• Multiple Choice: RACE, DREAM, ReClor, Multi-RC.
• Span Extraction: Hotpot QA, SQuAD 2.0.

Construct the training samples with specific masking strategies:

python processors/wiki_en_mask_shuffle_wk5_mp.py \
    --input_file <path_to_wiki_data> \
    --seed 42 \
    --keep_prob 0.7 \
    --ratio "(0.8,0.1,0.3,0.0,0.0,0.0)"

python main_wiki_pretrain.py --config <path_to_config_json>

Execute the provided bash scripts for specific tasks:

# Example for RACE
bash scripts/race/bert_iter_sr_mlm_v1.sh

# Example for Multi-RC
python main_multirc_torch151.py --config configs/multirc/mlm_baseline/bert-base-sc-v1-mlm-2-0-20k.json

@inproceedings{jiao-etal-2021-rept,
    title = "{REPT}: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training",
    author = "Jiao, Fangkai and Guo, Yangyang and Niu, Yilin and Ji, Feng and Li, Feng-Lin and Nie, Liqiang",
    booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
    month = aug,
    year = "2021",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.findings-acl.13",
    doi = "10.18653/v1/2021.findings-acl.13",
    pages = "150--163"
}

This work was supported by the National Key Research and Development Project of New Generation Artificial Intelligence and the Alibaba Research Intern Program.

This project is released under the terms of the LICENSE file included in this repository.