# if file is a txt file load it

if file.endswith(".txt"):

with open(os.path.join(labelled_file_path, file), "r") as f:

task = json.load(f)

raw_text = task['task']['data']['text']

annotation_result = task['result']

labelled_entities = [annotation['value'] for annotation in annotation_result]

# add the current text length to the start and end indices of labels plus one for the space

for entity in labelled_entities:

entity['start'] += len(combined_text)

entity['end'] += len(combined_text)

all_labelled_entities += labelled_entities

# add the current text to the combined text with space in between

combined_text += raw_text + " "

# copy the lists so they aren't modified outside this function

labelled_tokens = copy.deepcopy(labelled_tokens)

predicted_tokens = copy.deepcopy(predicted_tokens)

# considered a separate entity and update the token label objects

for i, document in enumerate(labelled_tokens):

document = [label.replace("I-", "B-") for label in document]

labelled_tokens[i] = document

for i, document in enumerate(predicted_tokens):

document = [label.replace("I-", "B-") for label in document]

predicted_tokens[i] = document

labelled_tokens, predicted_tokens, output_dict=True, zero_division=0

# copy the lists so they aren't modified outside this function

labelled_tokens = copy.deepcopy(labelled_tokens)

predicted_tokens = copy.deepcopy(predicted_tokens)

# considered a separate entity and update the token label objects

for i, document in enumerate(labelled_tokens):

document = [label.replace("I-", "B-") for label in document]

labelled_tokens[i] = document

for i, document in enumerate(predicted_tokens):

document = [label.replace("I-", "B-") for label in document]

predicted_tokens[i] = document

accuracy = accuracy_score(labelled_tokens, predicted_tokens)

f1 = f1_score(labelled_tokens, predicted_tokens)

recall = recall_score(labelled_tokens, predicted_tokens)

precision = precision_score(labelled_tokens, predicted_tokens)

Original code addopted from here on May 23, 2023: https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification

The following improvements were added to the tools linked above:

1. The ability to log models and metrics to an Azure ML workspace MLflow instance was added and requiring environment variables to be set and the azureml-mlflow package to be installed to log.

1. The environnment variables AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET and AZURE_MLFLOW_TRACKING_URI must be set using the .env file in the root of the repo.

2. Adding in the automated text preprocessing from labelstudio outputs in the `labelstudio_preprocessing.py` file which is added as a bash script target.

Fine-tuning the library models for token classification task such as Named Entity Recognition (NER), Parts-of-speech

tagging (POS) or phrase extraction (CHUNKS). The main scrip `run_ner.py` leverages the 🤗 Datasets library and the Trainer API. You can easily

customize it to your needs if you need extra processing on your datasets.

It will either run on a datasets hosted on our [hub](https://huggingface.co/datasets) or with your own text files for

training and validation, you might just need to add some tweaks in the data preprocessing.

The following example fine-tunes BERT on CoNLL-2003:

python run_ner.py \

--model_name_or_path bert-base-uncased \

--dataset_name conll2003 \

--output_dir /tmp/test-ner \

--do_train \

--do_eval

or just can just run the bash script `run.sh`.

To run on your own training and validation files, use the following command:

python run_ner.py \

--model_name_or_path bert-base-uncased \

--train_file path_to_train_file \

--validation_file path_to_validation_file \

--output_dir /tmp/test-ner \

--do_train \

--do_eval

**Note:** This script only works with models that have a fast tokenizer (backed by the 🤗 Tokenizers library) as it

uses special features of those tokenizers. You can check if your favorite model has a fast tokenizer in

[this table](https://huggingface.co/transformers/index.html#supported-frameworks), if it doesn't you can still use the old version

of the script.

You can find the old version of the PyTorch script [here](https://github.com/huggingface/transformers/blob/main/examples/legacy/token-classification/run_ner.py).

Based on the script [run_ner_no_trainer.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner_no_trainer.py).

Like `run_ner.py`, this script allows you to fine-tune any of the models on the [hub](https://huggingface.co/models) on a

token classification task, either NER, POS or CHUNKS tasks or your own data in a csv or a JSON file. The main difference is that this

script exposes the bare training loop, to allow you to quickly experiment and add any customization you would like.

It offers less options than the script with `Trainer` (for instance you can easily change the options for the optimizer

or the dataloaders directly in the script) but still run in a distributed setup, on TPU and supports mixed precision by

the mean of the [🤗 `Accelerate`](https://github.com/huggingface/accelerate) library. You can use the script normally

after installing it:

pip install git+https://github.com/huggingface/accelerate

then

export TASK_NAME=ner

python run_ner_no_trainer.py \

--model_name_or_path bert-base-cased \

--dataset_name conll2003 \

--task_name $TASK_NAME \

--max_length 128 \

--per_device_train_batch_size 32 \

--learning_rate 2e-5 \

--num_train_epochs 3 \

--output_dir /tmp/$TASK_NAME/

You can then use your usual launchers to run in it in a distributed environment, but the easiest way is to run

accelerate config

and reply to the questions asked. Then

accelerate test

that will check everything is ready for training. Finally, you can launch training with

export TASK_NAME=ner

accelerate launch run_ner_no_trainer.py \

--model_name_or_path bert-base-cased \

--dataset_name conll2003 \

--task_name $TASK_NAME \

--max_length 128 \

--per_device_train_batch_size 32 \

--learning_rate 2e-5 \

--num_train_epochs 3 \

--output_dir /tmp/$TASK_NAME/

This command is the same and will work for:

- a CPU-only setup

- a setup with one GPU

- a distributed training with several GPUs (single or multi node)

- a training on TPUs

Note that this library is in alpha release so your feedback is more than welcome if you encounter any problem using it.