How to perform function/tool calling using non-OpenAI models.

[niuhuluzhihao]

Assuming that I customize the tool function, must my model have the ability to tool calling? If my model does not have the tool_calling capability, how should I embed the tool into my custom agent?

[ekzhu]

Currently for tool calling it requires LLM support. But you can use code generation capability of LLM. You would need to customize the system_message to include the definition of the function.

[niuhuluzhihao]

Thank you very much for your answer.
But my tools may contain interfaces for calling other microservices. I have an immature idea: refer to
self.register_reply([Agent, None], MathUserProxyAgent._generate_math_reply, position=2)
in 'MathUserProxyAgent'.
I define my own 'ToolSUserProxyAgent' and my own '_generate_tool_reply', and implement the method of calling other microservices in '_generate_tool_reply'.
I'm not sure if this is possible?Could you help me?

[ekzhu]

If the LLM at least support generating structured response i.e., a JSON object corresponding to the remote API service call, then you should be able to prompt it to generate that given a natural language task. This is for the Assistant Agent.

For the UserProxyAgent, you can register a new reply function like what you mentioned, and, in that function, you parse the JSON object and make the actual call to the remote API service, and then pass the response back to the Assistant Agent.

Then you call user_proxy_agent.initiate_chat(assistant_agent, "<your task description>") to start the conversation.

The hardest part in my opinion is the getting the system prompt right for the assistant agent since it is not an OpenAI model. Depending on the model maybe you also need to give some examples as in-context learning.

[niuhuluzhihao]

Thank you. You are right. Since it is not an openai model, it does have insufficient capabilities. Setting the system prompt is indeed difficult and may require many iterations. I will try it.

In addition, in my opinion, for my Agent that calls the remote API, registering new functions seems more like customization and does not seem flexible enough.I will consider it.

[ekzhu]

Yes it is not convenient. Maybe a good way is to provide a library of working structured response generators using something like Guidance. See #490. So you can have that wraps around the model to enforce the generation. So you don’t have to parse potentially hairy unstructured data in user proxy. Perhaps you can try it out and create an issue or a PR for that.

[niuhuluzhihao]

Thank you.I just read https://github.com/microsoft/autogen/blob/main/notebook/agentchat_stream.ipynb which seems to be about calling external tools through registration.

I haven't read the question #490 yet. When I finish reading it, I will move on to my next step.