Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cocoteros_es task in spanish_bench #2721

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sgs97ua
Copy link

@sgs97ua sgs97ua commented Feb 21, 2025

  • Add lm_eval/tasks/spanish_bench/cocoteros_es.yaml task.
  • Update task lm_eval/tasks/spanish_bench/README.md.
  • Update file lm_eval/tasks/spanish_bench/spanish_bench.yaml.

Dataset: Cocoteros Hugging Face.
Paper: https://besaya.infor.uva.es/sepln24/paper04.pdf

The dataset was tested with the following command:

python -m scripts.write_out --sets test --output_base_path "testhm" --tasks "cocoteros_es"

Output of previous command:

!!@@##@@!! -- Example 0 Genera una frase corta con estas palabras: ['morder', 'perro', 'mano']. El contexto es: Una mujer estaba paseando a su perro cuando el animal se asustó y se escapó. El perro corrió hacia un grupo de personas, y una de ellas resultó herida.   Respuesta: Fue ese perro el que me mordió la mano.  Genera una frase corta con estas palabras: ['cepillo', 'diente', 'comida']. El contexto es: La higiene bucodental es clave para prevenir enfermedades dentales.   Respuesta:

The task was evaluated obtaining the following results:

Model BLEU ROUGE1
LLama 3 8B 17,1442 0,4547
Salamabra 7B 7,8484 0,3292

@CLAassistant
Copy link

CLAassistant commented Feb 21, 2025

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants