Hi team,
I am using evalplus v0.3.1 and running the evaluation for mbpp benchmark. However, it consistenly getting stuck and the timeout does not seem to work https://github.com/evalplus/evalplus/blob/master/evalplus/eval/__init__.py#L265. I suspect it is due to Mbpp/255 and some other similar questions where the generated solution includes something like ```
def combinations_colors(colors, n):\n import itertools\n return list(itertools.product(colors, repeat=n))
Not sure why python process won't stop even with the timeout. Here is my sample to reproduce this problem https://drive.google.com/file/d/1tM8dbV_jehjSGsXMIkOeznjzAfAaogLc/view?usp=sharing
Hi team,
I am using evalplus v0.3.1 and running the evaluation for mbpp benchmark. However, it consistenly getting stuck and the timeout does not seem to work https://github.com/evalplus/evalplus/blob/master/evalplus/eval/__init__.py#L265. I suspect it is due to Mbpp/255 and some other similar questions where the generated solution includes something like ```
def combinations_colors(colors, n):\n import itertools\n return list(itertools.product(colors, repeat=n))