Create evals

[homanp]

Would be nice to have an eval pipeline to continuously measure performance/cost gains over a set of LLMs and documents.

[g-hano]

are you planning to include Ollama models in the evaluation? Since their performance metrics would depend heavily on local hardware, we might need a different approach compared to API-based models

[homanp]

are you planning to include Ollama models in the evaluation? Since their performance metrics would depend heavily on local hardware, we might need a different approach compared to API-based models

I was thinking starting with the hosted models, since as you correctly point out local inference would need it's own evals.

The tricky part here is to find a good dataset that is accepted as a benchmark. Any ideas?

[g-hano]

After a quick search, I found Frames and RewardBench

[homanp]

After a quick search, I found Frames and RewardBench

I would go with frames