add support for some models whose tool call arguments is not a json object

[htao7]

What feature would you like to be added?

Some of the models' tool call arguments does not return a json object but a dictionary.

For example, AssistantAgent using DeepSeek-R1-Distill-Qwen-1.5B on HF inference api, inner_messages:
[ToolCallRequestEvent(source='Weather_Agent', models_usage=RequestUsage(prompt_tokens=173, completion_tokens=19), content=[FunctionCall(id='0', arguments={'location': 'Hangzhou'}, name='get_weather')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='Weather_Agent', models_usage=None, content=[FunctionExecutionResult(content='Error: the JSON object must be str, bytes or bytearray, not dict', call_id='0')], type='ToolCallExecutionEvent')]

As a reference, gpt-4o-mini inner_messages:
[ToolCallRequestEvent(source='Weather_Agent', models_usage=RequestUsage(prompt_tokens=69, completion_tokens=16), content=[FunctionCall(id='call_xxx', arguments='{"location":"Hangzhou"}', name='get_weather')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='Weather_Agent', models_usage=None, content=[FunctionExecutionResult(content='Hangzhou, nice!', call_id='call_6F0V1emkKJCleuTmo0PKpzge')], type='ToolCallExecutionEvent')]

Add support for dictionary type argument can allow the usage of these models. A temporary fix (for dict only) could be done at autogen_agentchat/agents/_assistant_agent.py:

async def _execute_tool_call(
        self, tool_call: FunctionCall, cancellation_token: CancellationToken
    ) -> FunctionExecutionResult:
        """Execute a tool call and return the result."""
        try:
            if not self._tools + self._handoff_tools:
                raise ValueError("No tools are available.")
            tool = next((t for t in self._tools + self._handoff_tools if t.name == tool_call.name), None)
            if tool is None:
                raise ValueError(f"The tool '{tool_call.name}' is not available.")
            if isinstance(tool_call.arguments, dict):
                arguments = tool_call.arguments
            else:
                arguments = json.loads(tool_call.arguments)
            result = await tool.run_json(arguments, cancellation_token)
            result_as_str = tool.return_value_as_string(result)
            return FunctionExecutionResult(content=result_as_str, call_id=tool_call.id)
        except Exception as e:
            return FunctionExecutionResult(content=f"Error: {e}", call_id=tool_call.id)

will have inner_messages:
[ToolCallRequestEvent(source='Weather_Agent', models_usage=RequestUsage(prompt_tokens=173, completion_tokens=19), content=[FunctionCall(id='0', arguments={'location': 'Hangzhou'}, name='get_weather')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='Weather_Agent', models_usage=None, content=[FunctionExecutionResult(content='Hangzhou, nice!', call_id='0')], type='ToolCallExecutionEvent')]

Why is this needed?

support for some models whose tool call is not a json object, e.g., DeepSeek on HF

[ekzhu]

Hello @htao7 !

Can you post your code including the model client? What model client did you use?

It seems strange because a model client shouldn't be returning the arguments as a dictionary. It looks like a bug in the model client.

[htao7]

Hello @htao7 !

Can you post your code including the model client? What model client did you use?

It seems strange because a model client shouldn't be returning the arguments as a dictionary. It looks like a bug in the model client.

I tested using deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B from HF inference API (haven't tested with DeepSeek API directly since it is under maintenance). I believe it is openAI compatible?

from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_core.tools import FunctionTool
from dotenv import load_dotenv
import os

load_dotenv()

model_client = OpenAIChatCompletionClient(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
    base_url="https://api-inference.huggingface.co/v1/",
    api_key=os.getenv("hf_token"),
    timeout=30,
    model_info={
        "vision": False,
        "function_calling": True,
        "json_output": True,
    },
)

def get_weather(location: str) -> str:
    return(f'{location}, nice!')

weather_tool = FunctionTool(
    get_weather, description="Search weather for a location"
)

from autogen_agentchat.agents import AssistantAgent

weather_agent = AssistantAgent(
    name="Weather_Agent",
    tools=[weather_tool],
    model_client=model_client,
    description="An agent that can search for weather.",
    system_message="You are a helpful AI assistant. Solve tasks using your tools.",
)

from autogen_agentchat.messages import TextMessage
from autogen_core import CancellationToken


async def assistant_run() -> None:
    response = await weather_agent.on_messages(
        [TextMessage(content="search on weather of Hangzhou", source="user")],
        cancellation_token=CancellationToken(),
    )
    print(response.inner_messages, '\n')
    # print(response.chat_message)

await assistant_run()

[ekzhu]

Looks like Hugging Face API is not strictly openai-compatible. Add a PR #5260 to fix this.