This repository is the official implementation of DrafterBench. We provide evaluation data, codes, and a brief introduction.

The DrafterBench is designed to evaluate large language models (LLMs) as an agent to automate monotonous, low-tech, and high-labor-intensity tasks in industry. Our initiative is drawing revision, which is a representation task in civil engineering that urgently needs to be automated. We took the following workflow to simulate the working scenario and evaluate the strengths and limitations of LLMs as automation agents.

After the stage of preprocessing, the drawing revision tasks (summarized from the real world, totalling 1920 across 12 types) are converted into natural language processing (NLP) tasks to evaluate complex function calls instructed by intricate and lengthy content commands. We designed over 40 drawing revision tools and provided them to LLMs, which play different functions. Some of them aim to make visible changes to drawings, while the others serve necessary preparations for them (e.g., opening the file or transferring critical arguments). It's difficult to determine whether the tools called are effective and functioning properly from the revised drawings, especially when checking if there are redundant or duplicated invisible tools. Therefore, to accurately evaluate the models‘ performance, we score their responses based on the operation chains rather than the revised drawing results.

To record the operation chains, we prepared dual functions for the tools provided to the LLMs. Each dual function has the same name, input, and output type as the original tools, and its function is to capture the operations and valuable data in a well-structured JSON format (e.g., argument value, data type, etc.). During the working of the benchmark, the original tools called by the models will be replaced with dual functions to record the operation chains and help the final assessment.

There are four essential capabilities evaluated by DrafterBench:

• Structured data understanding
• Function execution
• Instruction following
• Critical reasoning

• Dataset Summary
• Quick Start
• LeaderBoard

The DrafterBench is constructed on tasks over three object elements, four operations, and six complexity controllers.

The dataset is available here on Huggingface.

First, configure an environment with Python 3.11 and download the repositories.

git clone https://github.com/Eason-Li-AIS/DrafterBench.git
cd DrafterBench

Then, install the dependencies.

pip install -e .

• For API calling, set up your OpenAI / Anthropic / Google / Mistral / Deepinfra / AnyScale or other API keys as environment variables.

  OPENAI_API_KEY=...
  ANTHROPIC_API_KEY=...
  GOOGLE_API_KEY=...
  MISTRAL_API_KEY=...
  DEEPINFRA_API_KEY=...
  HUGGINGFACE_TOKEN=...

• For customized model, provide your vllm url when running evaluation.py

  --vllm_url http://xx.xx.xx.xx:8000/v1

Specify the --model and --model-provider flags to run DrafterBench. The supported models and providers are available here. You can name your experiment with the --exp_name flag, or it will be set as "model+time+task_group" by default.

python evaluation.py --model gpt-4o-2024-08-06 --model-provider openai --temperature 0.0

• To run tasks of a specific set, use the --task_group flag. You can choose each set in ["Structured", "Unstructured", "Precise", "Vague", "Complete", "Error", "Single_Object", "Multiple_Objects", "Single_Operation", "Multiple_Operations"]. For example:

  python evaluation.py --model gpt-4o-2024-08-06 --model-provider openai --task_group Structured

  This command will run only the tasks in a structured language. The default task group is "All" tasks.

• To have a clear view of the result, you can set up your huggingface token,

   HUGGINGFACE_TOKEN=...

  then use the --huggingface_user_name flag to provide your Huggingface user name. Our benchmark will create a new dataset repository with the --exp_name and push the results to it. This repository is private by default, you can create a public repository by setting the --huggingface_private flag to False.

  python evaluation.py --model gpt-4o-2024-08-06 --model-provider openai --task_group Structured --huggingface_user_name XXXXX(Replace "XXXXX" with your Huggingface username)

• The default prompts for 12 tasks can be found in ./prompts. You are encouraged to develop your own prompts to achieve a higher score. To do so, simply replace the default prompts in .txt file with your new prompts.

• In case the evaluation is unexpectedly interrupted, DrafterBench supports resuming from existing results. You can specify the result file for resuming in the --resume_from flag. Alternatively, you can set the --auto_resume flag to True, and DrafterBench will automatically search the result directory for the latest file that matches the model name and task group, and resume the remaining evaluation process.

  python evaluation.py --model gpt-4o-2024-08-06 --model-provider openai --task_group Structured --resume_from *****.json

  python evaluation.py --model gpt-4o-2024-08-06 --model-provider openai --task_group Structured --auto_resume True

Note: We have recently upgraded DrafterBench to be more challenging. Although the trend of models' ability is very consistent with the above leaderboard, some models may score lower than the records. The score on the leaderboard is the average of three independent runs.

If you use DrafterBench in your research, please consider citing our paper:

@article{drafterbench,
  title={DrafterBenchmark: Benchmarking Large Language Models for Tasks Automation in Civil Engineering},
  author={Yinsheng Li, Zhen Dong, Yi Shao.},
  year={2025},
  url={https://arxiv.org/abs/2507.11527},
}