Question about humaneval

[Shiguang-Guo]

I tried to evaluate humaneval on meta-llama-3.1-instruct, but got a score close to 0. I printed the output and found

{
    "resps": [
        [
            "Here's a Python function that implements the required functionality:\n\n```python\nfrom typing import List\n"
        ]
    ]
}

I think this may be due to generation_kwargs.until in the configuration. So what is the correct way to evaluate?

[baberabb]

Hi! what model are you using?

[Shiguang-Guo]

I use Meta-Llama-3.1-8B-Instruct and run with

lm_eval --model vllm --model_args="pretrained=${model_name},dtype=auto,tensor_parallel_size=${GPUS_PER_NODE},max_model_len=16384,gpu_memory_utilization=0.9,enable_chunked_prefill=True" --tasks=humaneval --batch_size=auto --output_path results --apply_chat_template --fewshot_as_multiturn --gen_kwargs="stop_token_ids=[128009]"

[baberabb]

Looks like a prompting/answer extraction issue. Added the prompt from llama evals (as humaneval_instruct) in the PR, but score still lower then official (0.5976 vs. 0.726 )