How can i get the answers in the dataset?

[Guozhenyuan]

The CodeMMLU is a great piece of work!

I noticed that the dataset provides task_id, question, and choices columns, but is there an answer column?

How should I handle this dataset if I want to fine-tune an LLM?

[sinsauzero]

+1

[nmd2k]

Thank you for your interest in CodeMMLU! @Guozhenyuan @sinsauzero

1. In order to evaluate with CodeMMLU, you can hand me the LLM's response via email (follow format mentioned in HERE).

2. Since CodeMMLU is intended as a benchmark, not a training dataset, its purpose is to evaluate the performance of pre-trained LLMs, so it might not correctly serve fine-tune purpose.