Skip to content

Commit

Permalink
Leaderboard Update, in sync with BFCL April 9th Release (#341)
Browse files Browse the repository at this point in the history 
This PR updates the leaderboard data, as mentioned in #338. As a result,
some values/scores are changed.
Note that the model `glaiveai/glaive-function-calling-v1` is excluded in
this leaderboard update PR due to the model's tokenizer problem. We
cannot generate that model's output data. So to avoid confusion, we
excluded that model in this update.
  • Loading branch information
@HuanzhiMao
HuanzhiMao authored Apr 11, 2024
1 parent 94cee0a commit 06c7f9d
Show file tree
Hide file tree
Showing 3 changed files with 31 additions and 31 deletions.
58 changes: 29 additions & 29 deletions data.csv
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
Rank,Overall Acc,Model,Model Link,Organization,License,AST Summary,Exec Summary,Simple Function AST,Python Simple Function AST,Java Simple Function AST,JavaScript Simple Function AST,Multiple Functions AST,Parallel Functions AST,Parallel Multiple AST,Simple Function Exec,Python Simple Function Exec,REST Simple Function Exec,Multiple Functions Exec,Parallel Functions Exec,Parallel Multiple Exec,Relevance Detection,Cost ($ Per 1k Function Calls),Latency Mean (s),Latency Standard Deviation (s)
1,80.47%,Claude-3-Opus-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,80.30%,69.57%,84.18%,90.25%,65.00%,74.00%,92.00%,81.00%,64.00%,85.29%,87.00%,82.86%,76.00%,72.00%,45.00%,80.83%,10.81,5.67,1.43
2,79.94%,GPT-4-0125-Preview (Prompt),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,82.48%,69.30%,84.91%,90.00%,67.00%,80.00%,90.50%,87.50%,67.00%,84.71%,86.00%,82.86%,78.00%,72.00%,42.50%,69.17%,5.22,2.1,1.4
3,78.88%,Gorilla-OpenFunctions-v2 (FC),https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html,Gorilla LLM,Apache 2.0,82.41%,70.93%,83.64%,93.50%,53.00%,66.00%,93.50%,85.50%,67.00%,88.24%,85.00%,92.86%,78.00%,70.00%,47.50%,60.83%,1.53,2.38,2.05
4,78.76%,GPT-4-0125-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,81.58%,65.17%,79.82%,89.75%,56.00%,48.00%,91.50%,89.00%,66.00%,71.18%,84.00%,52.86%,70.00%,72.00%,47.50%,81.67%,4.76,4.37,5.31
5,78.53%,GPT-4-1106-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,79.56%,68.40%,78.73%,88.50%,49.00%,60.00%,88.50%,88.50%,62.50%,80.59%,83.00%,77.14%,76.00%,72.00%,45.00%,80.83%,4.95,6.45,6.45
6,74.71%,Claude-3-Sonnet-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,80.00%,67.37%,80.00%,89.00%,52.00%,64.00%,86.00%,87.00%,67.00%,76.47%,76.00%,77.14%,76.00%,72.00%,45.00%,53.33%,2.13,2.38,0.66
7,74.29%,Functionary-Medium-v2.4 (FC),https://huggingface.co/meetkai/functionary-medium-v2.4,MeetKai,MIT,77.14%,61.96%,78.55%,85.75%,57.00%,64.00%,88.50%,80.00%,61.50%,58.82%,80.00%,28.57%,68.00%,76.00%,45.00%,75.42%,1.64,2.56,2.51
8,73.06%,Functionary-Small-v2.4 (FC),https://huggingface.co/meetkai/functionary-small-v2.4,MeetKai,MIT,75.38%,64.14%,78.00%,86.00%,54.00%,62.00%,83.00%,80.50%,60.00%,67.06%,81.00%,47.14%,66.00%,76.00%,47.50%,67.50%,1.76,2.74,2.55
9,72.88%,Mistral-Medium-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,71.10%,58.08%,74.91%,83.00%,54.00%,52.00%,74.00%,80.00%,55.50%,68.82%,79.00%,54.29%,70.00%,66.00%,27.50%,88.33%,1.75,3.41,4.65
10,70.29%,Claude-instant-1.2 (Prompt),https://www.anthropic.com/news/releasing-claude-instant-1-2,Anthropic,Proprietary,72.27%,63.46%,77.09%,84.50%,55.00%,62.00%,85.00%,79.50%,47.50%,78.82%,85.00%,70.00%,68.00%,62.00%,45.00%,54.17%,0.95,1.75,1.0
11,69.53%,Claude-3-Opus-20240229 (FC),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,70.82%,63.76%,71.27%,82.00%,40.00%,48.00%,75.50%,76.50%,60.00%,77.06%,83.00%,68.57%,68.00%,70.00%,40.00%,62.50%,26.58,14.82,6.0
12,66.71%,Mistral-Small-2402 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,74.23%,30.07%,76.91%,83.50%,58.00%,62.00%,84.00%,81.00%,55.00%,25.29%,1.00%,60.00%,8.00%,62.00%,25.00%,76.25%,1.04,1.62,1.75
13,65.00%,Claude-3-Sonnet-20240229 (FC),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,67.70%,60.55%,69.82%,79.25%,38.00%,58.00%,71.50%,73.50%,56.00%,64.71%,85.00%,35.71%,70.00%,70.00%,37.50%,51.67%,5.08,6.7,3.07
14,64.24%,Claude-2.1 (Prompt),https://www.anthropic.com/news/claude-2-1,Anthropic,Proprietary,60.10%,46.48%,74.91%,80.75%,59.00%,60.00%,73.50%,56.00%,36.00%,49.41%,63.00%,30.00%,68.00%,36.00%,32.50%,83.33%,6.62,3.2,1.42
15,63.29%,Claude-3-Haiku-20240307 (FC),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,72.86%,59.57%,75.45%,85.00%,51.00%,48.00%,77.00%,77.00%,62.00%,51.76%,84.00%,5.71%,74.00%,70.00%,42.50%,21.67%,0.44,3.81,2.32
16,63.12%,Mistral-large-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,58.19%,48.62%,65.27%,88.50%,3.00%,4.00%,89.00%,23.00%,55.50%,70.00%,77.00%,60.00%,74.00%,8.00%,42.50%,84.17%,4.74,2.91,2.73
17,61.59%,DBRX-Instruct (Prompt),https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm,Databricks,Databricks Open Model,63.55%,60.54%,62.18%,73.00%,26.00%,48.00%,73.50%,73.00%,45.50%,61.18%,79.00%,35.71%,74.00%,62.00%,45.00%,54.58%,1.24,0.62,0.41
18,58.35%,Mistral-large-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,64.61%,52.17%,79.45%,87.50%,58.00%,58.00%,90.50%,32.50%,56.00%,81.18%,85.00%,75.71%,76.00%,4.00%,47.50%,0.00%,3.86,1.99,2.53
19,57.65%,GPT-3.5-Turbo-0125 (FC),https://platform.openai.com/docs/models/gpt-3-5-turbo,OpenAI,Proprietary,67.41%,68.30%,57.64%,58.75%,52.00%,60.00%,67.00%,87.50%,57.50%,84.71%,80.00%,91.43%,72.00%,74.00%,42.50%,2.08%,0.42,1.53,1.25
20,56.35%,Gemini-1.0-Pro (FC),https://deepmind.google/technologies/gemini/#introduction,Google,Proprietary,41.48%,39.33%,76.91%,88.25%,43.00%,54.00%,89.00%,0.00%,0.00%,78.82%,84.00%,71.43%,76.00%,0.00%,2.50%,76.67%,0.18,1.05,0.23
21,55.76%,Mistral-small-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,60.07%,47.17%,79.27%,87.25%,57.00%,60.00%,91.50%,37.00%,32.50%,81.18%,86.00%,74.29%,78.00%,12.00%,17.50%,0.00%,0.95,1.14,1.15
22,51.53%,Nexusflow-Raven-v2 (FC),https://huggingface.co/Nexusflow/NexusRaven-V2-13B,Nexusflow,Apache 2.0,56.91%,57.74%,67.64%,73.25%,49.00%,60.00%,83.00%,35.00%,42.00%,56.47%,80.00%,22.86%,74.00%,58.00%,42.50%,2.08%,N/A,1.9,1.57
23,50.29%,FireFunction-v1 (FC),https://huggingface.co/fireworks-ai/firefunction-v1,Fireworks,Apache 2.0,37.90%,32.99%,67.09%,86.00%,14.00%,22.00%,84.50%,0.00%,0.00%,62.94%,75.00%,45.71%,64.00%,0.00%,5.00%,73.33%,N/A,0.92,0.8
24,48.53%,GPT-4-0613 (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,37.05%,25.90%,60.18%,81.75%,3.00%,2.00%,88.00%,0.00%,0.00%,40.59%,57.00%,17.14%,58.00%,0.00%,5.00%,90.83%,10.27,3.64,3.99
25,47.29%,Mistral-tiny-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,43.12%,27.76%,50.00%,61.75%,28.00%,0.00%,49.00%,46.00%,27.50%,23.53%,39.00%,1.43%,28.00%,42.00%,17.50%,84.17%,0.13,1.14,1.19
26,39.59%,Gemma-7b-it (Prompt),https://blog.google/technology/developers/gemma-open-models/,Google,gemma-terms-of-use,38.32%,28.57%,41.27%,46.50%,29.00%,24.00%,49.00%,29.50%,33.50%,25.29%,39.00%,5.71%,34.00%,30.00%,25.00%,57.08%,0.03,0.09,N/A
27,37.47%,Deepseek-v1.5 (Prompt),https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5,Deepseek,Deepseek License,33.73%,30.85%,36.91%,45.75%,7.00%,26.00%,41.50%,33.00%,23.50%,35.88%,41.00%,28.57%,40.00%,30.00%,17.50%,56.25%,0.45,1.2,N/A
28,30.00%,Glaive-v1 (FC),https://huggingface.co/glaiveai/glaive-function-calling-v1,Glaive,cc-by-sa-4.0,29.02%,0.00%,53.09%,71.25%,3.00%,8.00%,63.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,38.33%,0.02,0.06,N/A
29,18.12%,Mistral-small-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,2.86%,7.79%,1.45%,2.00%,0.00%,0.00%,3.50%,4.00%,2.50%,21.18%,19.00%,24.29%,10.00%,0.00%,0.00%,99.58%,1.99,2.82,1.43
1,83.82%,GPT-4-0125-Preview (Prompt),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,88.75%,70.07%,88.00%,94.25%,67.00%,80.00%,94.50%,90.50%,82.00%,85.29%,86.00%,84.29%,78.00%,72.00%,45.00%,70.42%,5.21,1.99,1.27
2,83.53%,Claude-3-Opus-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,86.09%,69.43%,86.36%,93.25%,66.00%,72.00%,93.50%,86.00%,78.50%,84.71%,87.00%,81.43%,76.00%,72.00%,45.00%,80.42%,10.8,5.05,1.78
3,81.24%,GPT-4-1106-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,84.75%,67.29%,82.00%,90.25%,61.00%,58.00%,91.00%,91.50%,74.50%,77.65%,85.00%,67.14%,72.00%,72.00%,47.50%,80.42%,5.03,6.34,6.04
4,81.12%,Gorilla-OpenFunctions-v2 (FC),https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html,Gorilla LLM,Apache 2.0,86.16%,70.05%,87.64%,94.25%,67.00%,76.00%,94.50%,87.50%,75.00%,84.71%,87.00%,81.43%,78.00%,70.00%,47.50%,60.83%,1.7,2.65,2.33
5,79.94%,GPT-4-0125-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,83.75%,65.25%,80.00%,90.25%,53.00%,52.00%,92.50%,90.00%,72.50%,70.00%,84.00%,50.00%,74.00%,72.00%,45.00%,82.92%,4.82,5.03,5.88
6,78.94%,Mistral-Medium-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,81.44%,61.16%,79.27%,88.50%,54.00%,56.00%,92.50%,84.00%,70.00%,67.65%,86.00%,41.43%,72.00%,70.00%,35.00%,88.75%,1.75,2.77,2.27
7,77.29%,Claude-3-Sonnet-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,85.20%,69.35%,81.82%,90.25%,53.00%,72.00%,88.50%,88.00%,82.50%,75.88%,75.00%,77.14%,78.00%,76.00%,47.50%,50.42%,2.12,2.11,1.15
8,76.88%,Functionary-Medium-v2.4 (FC),https://huggingface.co/meetkai/functionary-medium-v2.4,MeetKai,MIT,82.36%,62.38%,79.45%,88.00%,55.00%,60.00%,90.00%,87.50%,72.50%,60.00%,80.00%,31.43%,66.00%,76.00%,47.50%,74.17%,1.64,2.55,2.6
9,75.94%,Functionary-Small-v2.4 (FC),https://huggingface.co/meetkai/functionary-small-v2.4,MeetKai,MIT,80.00%,64.73%,80.00%,88.75%,56.00%,58.00%,89.00%,82.00%,69.00%,69.41%,81.00%,52.86%,68.00%,74.00%,47.50%,67.92%,1.76,2.74,2.47
10,72.65%,Claude-instant-1.2 (Prompt),https://www.anthropic.com/news/releasing-claude-instant-1-2,Anthropic,Proprietary,76.63%,63.20%,80.00%,87.25%,56.00%,70.00%,86.00%,83.00%,57.50%,75.29%,84.00%,62.86%,72.00%,58.00%,47.50%,54.17%,0.95,1.35,0.62
11,70.76%,Claude-3-Opus-20240229 (FC),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,73.43%,62.02%,72.73%,84.25%,39.00%,48.00%,80.00%,75.00%,66.00%,70.59%,80.00%,57.14%,70.00%,70.00%,37.50%,65.00%,26.31,13.56,5.21
12,66.53%,Claude-3-Sonnet-20240229 (FC),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,71.38%,62.10%,70.00%,79.50%,39.00%,56.00%,73.50%,76.50%,65.50%,59.41%,85.00%,22.86%,72.00%,72.00%,45.00%,51.67%,5.08,6.44,2.41
13,65.35%,Claude-3-Haiku-20240307 (FC),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,76.32%,59.07%,77.27%,86.75%,51.00%,54.00%,79.50%,78.50%,70.00%,51.76%,82.00%,8.57%,72.00%,70.00%,42.50%,22.50%,0.45,3.45,1.58
14,65.12%,Claude-2.1 (Prompt),https://www.anthropic.com/news/claude-2-1,Anthropic,Proprietary,62.59%,46.39%,74.36%,80.75%,54.00%,64.00%,75.50%,55.50%,45.00%,47.06%,63.00%,24.29%,66.00%,40.00%,32.50%,83.33%,6.64,3.72,2.02
15,64.59%,Mistral-large-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,62.09%,46.32%,66.36%,88.75%,5.00%,10.00%,94.00%,25.50%,62.50%,65.29%,76.00%,50.00%,76.00%,4.00%,40.00%,84.17%,4.94,2.84,2.57
16,64.24%,DBRX-Instruct (Prompt),https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm,Databricks,Databricks Open Model,65.31%,63.22%,66.73%,79.50%,30.00%,38.00%,72.00%,72.00%,50.50%,65.88%,76.00%,51.43%,70.00%,72.00%,45.00%,56.25%,1.25,0.63,0.45
17,61.12%,Mistral-large-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,68.98%,50.99%,82.91%,91.50%,62.00%,56.00%,93.00%,31.50%,68.50%,82.94%,84.00%,81.43%,74.00%,2.00%,45.00%,0.00%,3.9,1.86,1.09
18,58.35%,GPT-3.5-Turbo-0125 (FC),https://platform.openai.com/docs/models/gpt-3-5-turbo,OpenAI,Proprietary,70.52%,66.33%,57.09%,57.50%,53.00%,62.00%,65.50%,90.00%,69.50%,78.82%,81.00%,75.71%,70.00%,74.00%,42.50%,2.08%,0.42,1.26,0.68
19,58.12%,Mistral-small-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,64.27%,46.94%,81.09%,90.25%,56.00%,58.00%,95.50%,39.00%,41.50%,81.76%,85.00%,77.14%,74.00%,12.00%,20.00%,0.00%,0.96,1.05,0.95
20,56.41%,Gemini-1.0-Pro (FC),https://deepmind.google/technologies/gemini/#introduction,Google,Proprietary,41.94%,38.57%,77.27%,89.75%,37.00%,58.00%,90.50%,0.00%,0.00%,75.29%,82.00%,65.71%,74.00%,0.00%,5.00%,77.50%,0.19,1.06,0.55
21,51.18%,FireFunction-v1 (FC),https://huggingface.co/fireworks-ai/firefunction-v1,Fireworks,Apache 2.0,39.94%,33.40%,67.27%,86.50%,13.00%,22.00%,92.50%,0.00%,0.00%,60.59%,79.00%,34.29%,68.00%,0.00%,5.00%,73.33%,N/A,1.24,1.2
22,51.06%,Nexusflow-Raven-v2 (FC),https://huggingface.co/Nexusflow/NexusRaven-V2-13B,Nexusflow,Apache 2.0,55.05%,57.22%,70.18%,76.00%,52.00%,60.00%,75.50%,30.50%,44.00%,55.88%,81.00%,20.00%,76.00%,52.00%,45.00%,2.08%,N/A,1.86,1.39
23,49.65%,GPT-4-0613 (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,38.53%,25.90%,61.64%,83.50%,4.00%,2.00%,92.50%,0.00%,0.00%,40.59%,58.00%,15.71%,58.00%,0.00%,5.00%,91.67%,10.48,3.54,3.37
24,48.71%,Mistral-tiny-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,46.91%,28.71%,49.64%,61.75%,26.00%,0.00%,56.50%,47.50%,34.00%,22.35%,36.00%,2.86%,20.00%,50.00%,22.50%,82.08%,0.13,1.79,1.63
25,41.47%,Gemma-7b-it (Prompt),https://blog.google/technology/developers/gemma-open-models/,Google,gemma-terms-of-use,39.05%,33.15%,42.18%,47.75%,29.00%,24.00%,48.00%,30.00%,36.00%,30.59%,45.00%,10.00%,32.00%,40.00%,30.00%,60.42%,0.03,0.09,N/A
26,39.00%,Deepseek-v1.5 (Prompt),https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5,Deepseek,Deepseek License,36.98%,28.24%,38.91%,49.50%,4.00%,24.00%,48.50%,37.00%,23.50%,32.94%,38.00%,25.71%,36.00%,34.00%,10.00%,56.67%,0.45,1.2,N/A
27,37.65%,Mistral-Small-2402 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,37.66%,27.93%,5.64%,5.75%,6.00%,4.00%,8.00%,78.50%,58.50%,24.71%,4.00%,54.29%,0.00%,62.00%,25.00%,98.33%,0.7,1.09,0.95
28,32.47%,Mistral-Large-2402 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,28.92%,14.88%,14.18%,16.50%,3.00%,18.00%,13.50%,33.50%,54.50%,20.00%,8.00%,37.14%,8.00%,14.00%,17.50%,91.25%,1.11,1.72,1.04
29,17.53%,Mistral-small-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,2.53%,6.97%,1.64%,2.25%,0.00%,0.00%,2.50%,3.00%,3.00%,15.88%,16.00%,15.71%,12.00%,0.00%,0.00%,99.58%,2.02,2.93,1.61
2 changes: 1 addition & 1 deletion leaderboard.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ <h2>Error Type Analysis</h2>
<a href="blogs/8_berkeley_function_calling_leaderboard.html#metrics">evaluation metrics</a> and insight blog (coming soon).
</p>

<div id="0e086fa1-13be-408b-a557-107d06325345" class="plotly-graph-div" style="height:100%; width:100%;"></div>
<div id="924d403a-cbb4-4d1c-9424-e9916cf83b69" class="plotly-graph-div" style="height:100%; width:100%;"></div>
</div>

<!-- API Explorer Section -->
Expand Down
2 changes: 1 addition & 1 deletion treemap_2.js
View file

Large diffs are not rendered by default.

0 comments on commit 06c7f9d

Please sign in to comment.