This repo trains a Spanish ESG-specific BERT starting from BETO via Domain-Adaptive Pre-Training (DAPT) and then fine-tunes on ESG downstream tasks (sentence classification / NER).

1. Install

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. Data layout

Place raw text files (one document per line or one big corpus) under your Drive and point DATA_DIR to it.

Recommended layout:

/MyDrive/esg_corpus/
  train.txt     # large corpus (one document per line)
  valid.txt     # small held-out file

3. Run DAPT (MLM)

python scripts/train_mlm.py --config configs/mvp.yaml

4. Resume on a bigger GPU

• Upload checkpoints/beto_esg_mvp to your cloud VM.
• Rerun train_mlm.py with a larger config (e.g., configs/a100_300m.yaml) and resume_from_checkpoint: checkpoints/beto_esg_mvp.

• Colab: mount Drive and set DATA_DIR: "/content/drive/MyDrive/esg_corpus".
• Kaggle: upload train.txt as a Kaggle Dataset and set DATA_DIR accordingly (e.g., /kaggle/input/esg-corpus).

configs/        # YAML configs for runs
scripts/        # training, preprocessing, eval
data_raw/       # (ignored) raw PDFs/HTML (store in Drive)
data_proc/      # processed text (ignored)
checkpoints/    # model checkpoints (ignored)
notebooks/      # Colab/Kaggle entrypoints
docs/           # model card, datasheets

• Consider DVC for tracking large data stored in Drive/S3, and Weights & Biases for experiment tracking.