Model introduction

When running the provided bash script with the specified configurations, successful generation is achieved. However, during the evaluation process, an error occurs as shown below:

File "/usr/local/lib/python3.10/site-packages/bigcodebench/evaluate.py", line 209, in evaluate
    passk = [int(k) for k in pass_k.split(",")]
AttributeError: 'tuple' object has no attribute 'split'

DATASET=bigcodebench
MODEL=baseline_llama_1b
BACKEND=vllm
NUM_GPU=2
SPLIT=complete
SUBSET=full
export E2B_API_KEY=""
bigcodebench.evaluate
--model $MODEL
--split $SPLIT
--execution "e2b"
--subset $SUBSET
--backend $BACKEND
--check_gt_only

Model URL

https://huggingface.co/meta-llama/Llama-3.1-8B

Additional instructions (Optional)

No response

Author

No

Security

• I confirm that the model is safe to run which does not contain any malicious code or content.

Integrity

• I confirm that the model comes from unique and original work and does not contain any plagiarism.