The approaches considered along with the rationale for their inclusion/rejection from development are outlined in @tbl-ner-approaches.

| :------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| Rule Based Models | This served as the baseline using regex to extract known entities but was not developed further due to the known issues with text quality due to OCR issues and infeasibility for entities like SITE. |

| <br/> | <br/> |

For the transformer based models two approaches were used for training, spaCy command line interface (CLI) [@spacy] and HuggingFace's Training application programming interface (API) [@huggingface]. Each have advantages and disadvantages which are outlined in @tbl-spacy-pros-cons.

+--------------------------+-----------------------------------------------------------------------------------------------------+------------------------------------------+

+==========================+=====================================================================================================+==========================================+

| spaCy Config Training | - Can integrate with any transformer hosted on HuggingFace | - Knowledge of bash scripting required |

| | - Prebuilt config scripts that require minimal changes | - Limited configuration options |

Using the Hugging Face training API multiple models were trained and evaluated. Each base model along with the hypothesis behind it's selection is outlined in @tbl-hf-model-hypoth.

| RoBERTa-base | One of the typically best performing models for NER [@roberta-ner-wang] |

| RoBERTa-large | A larger model than the base version with potential to learn more complex relationships with the downside of larger compute times. |

| BERT-multilanguage | The known OCR issues and scientific nature of the text may mean the larger vocabulary of this multi-language model may deal with issues better. |

| XLM-RoBERTa-base | Another cross language model (XLM) but using the RoBERTa base architecture and pre-training. |

| Specter2 | This model is BERT based and finetuned on 6M+ scientific articles with it’s own scientific vocabulary making it well suited to analyzing research articles. |

: Hugging Face Model Hypotheses {#tbl-hf-model-hypoth tbl-colwidths="[25,75]"}

Final hyper parameters used to train the models are outlined in @tbl-hf-train-hyperparams.

| Batch size | - Maximized to utilize all available GPU memory, 8 for RoBERTa based models |

| Learning Rate | - Initially 5e-5 was used and observed rapid over fitting with eval loss reaching a minimum around 2-4 epochs then increasing for the next 5-10 |

| | - Moved to 2e-5 as well as introducing gradient accumulation of 3 epochs to increase effective batch size |

| Learning Rate Scheduler | - All training was done with a linear learning rate scheduler which linearly decreases learning rate across epochs |

| Warmup Ratio | - How many steps of training to increase LR from 0 to LR, shown to improve with Adam optimizer - [@borealisai2023tutorial] Set to 10% initially |

: Hugging Face Model Training Hyperparameters {#tbl-hf-train-hyperparams tbl-colwidths="[30,70]"}

Using the spaCy CLI, the two models were trained and evaluated with each models advantages and disadvanctages outlined in @tbl-spacy-model-pros-cons.

+------------------+----------------------------------------------------------------------+------------------------------------------------------+

+==================+======================================================================+======================================================+

| RoBERTa-base | - State-of-the-art pretrained transformer for NLP tasks in English | - Computationally expensive to train and inference |

| | - Context rich embeddings | - Cannot fine-tune |

| **Parameters** | **Notes** |

|:----------------------------|:---------------------------------------------------------------------------------------------------------------------------|

| <br/> | <br/> |

| <br/> | <br/> |

| <br/> | <br/> |

| <br/> | <br/> |

| <br/> | <br/> |

| <br/> | <br/> |

: spaCy CLI Final Hyperparameters {#tbl-spacy-hyperparams tbl-colwidths="[30,70]"}

{#fig-review_tool_snap}

The Article Data Extraction pipeline is containerized using Docker and contains the entity extraction model within it. It will be run on the xDD servers as xDD is not legally allowed to send full text articles off their servers. The container accepts full text articles, extracts the entities, and outputs a single JSON object for each article. The JSON objects are combined with the article relevance prediction results and loaded into the Data Review Tool. @fig-deployment_pipeline depicts the work flow.

url={https://catalog.ldc.upenn.edu/LDC2013T19}

}

}

@article{inproceedings,

date = {2023-06-27}

}