Skip to content

Evalplus stuck for mbpp #294

@zb1439

Description

@zb1439

Hi team,

I am using evalplus v0.3.1 and running the evaluation for mbpp benchmark. However, it consistenly getting stuck and the timeout does not seem to work https://github.com/evalplus/evalplus/blob/master/evalplus/eval/__init__.py#L265. I suspect it is due to Mbpp/255 and some other similar questions where the generated solution includes something like ```
def combinations_colors(colors, n):\n import itertools\n return list(itertools.product(colors, repeat=n))

Not sure why python process won't stop even with the timeout. Here is my sample to reproduce this problem https://drive.google.com/file/d/1tM8dbV_jehjSGsXMIkOeznjzAfAaogLc/view?usp=sharing

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions