Different models on same tasks gives same results when cache is active

[salvatore-cipolla]

Executing the same task with same configs on two different models produces same identical resulting metrics when using cache. If I don't use cache I get the correct results (different metrics). Is cache working properly?

To reproduce the problem:

lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen2.5-3B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 1 \
    --limit 10 \
    --output_path output/test \
    --use_cache cache/cache.db

lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.2-3B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 1 \
    --limit 10 \
    --output_path output/test \
    --use_cache cache/cache.db

I'm using the latest version of the library installed directly from this repo.

[baberabb]

will have a look! probably a hashing error