- get all predictions `wget https://huggingface.co/datasets/OpenDevin/Devin-SWE-bench-output/raw/main/devin_swe_outputs.json`

See [`SWE-bench/README.md`](./SWE-bench/README.md) for more details on how to run SWE-Bench for evaluation.

FROM ubuntu:20.04

RUN apt-get update && \

apt-get install -y bash gcc git jq wget && \

apt-get clean && \

rm -rf /var/lib/apt/lists/*

RUN git config --global user.email "[email protected]"

RUN git config --global user.name "swebench"

RUN apt update && apt install -y build-essential

RUN useradd -ms /bin/bash swe-bench

USER swe-bench

WORKDIR /home/swe-bench

ENV PATH="/home/swe-bench/miniconda3/bin:${PATH}"

ARG PATH="/home/swe-bench/miniconda3/bin:${PATH}"

RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh \

&& mkdir ~/.conda \

&& bash miniconda.sh -b \

&& rm -f miniconda.sh

RUN conda --version

COPY environment.yml .

RUN conda env create -f environment.yml

RUN pip install datasets python-dotenv gitpython

RUN conda init bash

CMD ["/bin/bash"]

Work in-progress.

**TODOs**:

- [ ] Generate `predictions` files given an OpenDevin `Agent` implementation. We could borrow something from [devin's eval-harness implementation](https://github.com/CognitionAI/devin-swebench-results/tree/main/harness), for example, [how to generate `TestSpec`](https://github.com/CognitionAI/devin-swebench-results/blob/main/harness/scripts.py#L150-L160).

- [ ] Make sure the evaluation suite runs on all repos. I only tested on `matplotlib` so far, `scikit-learn` does not work for now (see [this issue](https://github.com/princeton-nlp/SWE-bench/issues/57))).

Currently, the docker container should be able to for running SWE-Bench. It was tested on `matplotlib`, but it requires further testing to make sure it works on other repositories. Currently, [it does not work for `scikit-learn`](https://github.com/princeton-nlp/SWE-bench/issues/57)).

cd evaluation/SWE-bench

./scripts/prepare_devin_swe_bench_data.sh

git clone https://github.com/xingyaoww/SWE-bench.git

./scripts/run_docker_interactive.sh

mkdir -p data/logs

mkdir -p data/testbeds

python SWE-bench/harness/run_evaluation.py \

--predictions_path data/predictions/devin_swe_outputs.json \

--swe_bench_tasks data/processed/swe-bench-test.json \

--log_dir data/logs \

--testbed data/testbeds \

--skip_existing \

--timeout 900 \

--verbose

You will see the command line outputs similar to this (if success):

Then you may interpret the results under `data/logs`, and interpret it following [this guide](https://github.com/princeton-nlp/SWE-bench/blob/main/tutorials/evaluation.md#-metrics).

name: swe-bench

dependencies:

- python=3.9

- pip

- pip:

- beautifulsoup4

- chardet

- ghapi

- GitPython

- python-dotenv

- requests

- rich

- transformers>=4.34.0

- conda-forge::gh

from datasets import load_dataset

import pandas as pd

dataset = load_dataset("princeton-nlp/SWE-bench")

test = dataset["test"].to_pandas()

test.to_json("data/processed/swe-bench-test.json", orient="records")

set -xeo pipefail

mkdir -p data/processed

python3 scripts/download_test_data.py

mkdir -p data/predictions

wget https://huggingface.co/datasets/OpenDevin/Devin-SWE-bench-output/raw/main/devin_swe_outputs.json -O data/predictions/devin_swe_outputs.json

DOCKER_IMAGE=opendevin/eval-swe-bench:v0.1

WORK_DIR=`pwd`

docker run \

-it \

--rm \

--user root \

--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \

-v $WORK_DIR:/swe-bench \

-w /swe-bench \

$DOCKER_IMAGE \

/bin/bash -c "usermod -u $(id -u) swe-bench && su swe-bench"