Assistants won't call functions, only User Proxy will

[tyler-suard-parker]

Describe the issue

Hello. I have one user proxy, with no functions registered to it. I have 3 different assistant agents, each one with their own functions registered. When I start my async chat, the user proxy sends a question to each of the assistant agents. The system prompt for each agent tells them to use a specific function to search on their own. Instead of doing that, each agent recommends a search function to the user proxy, which it of course does not have access to. How can I solve this issue? Using windows and GPT-4.

Steps to reproduce

self.user_proxy = autogen.UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=10,
is_termination_msg=self.is_termination_msg_func,
llm_config=self.llm_config_userproxy,
system_message="""End your last message with TERMINATE if the user's question has been answered at full satisfaction.
Otherwise, reply CONTINUE, with the reason why the task is not solved yet.
Please use CONTINUE sparingly, and only when you strongly feel that another search would retrieve more information that would
help answer the user's question.
DO NOT answer based on your own knowledge, ONLY answer using the data presented.
Remember, you may reply with TERMINATE, or CONTINUE with a reason why the task is not solved yet.
If an agent recommends a search to you, tell the agent to search themself, because you don't have access to any tunctions.""",
)

    if len(self.message_history) > 1:
        for message in self.message_history[:-1]:
            self.user_proxy.send(
                str(message), self.documents_searchbot, request_reply=False
            )
            self.user_proxy.send(
                str(message), self.pim_wcs_searchbot, request_reply=False
            )
            self.user_proxy.send(
                str(message), self.crossref_searchbot, request_reply=False
            )

    self.documents_searchbot.register_function(
        function_map={
            "docs_search": self.exec_product_docs_search,
        }
    )
    self.pim_wcs_searchbot.register_function(
        function_map={
            "pim_wcs_search": self.exec_pim_wcs_search,
        }
    )
    self.crossref_searchbot.register_function(
        function_map={
            "crossref_acs": self.exec_crossref_search,
        }
    )

async def run_agents(self):
    async with aiohttp.ClientSession() as self.session:

        agent_chat_start = time.time()

        chat_results = await self.user_proxy.a_initiate_chats(
            [
                {
                    "chat_id": 1,
                    "recipient": self.documents_searchbot,
                    "message": {"content": str(self.message_history[-1])},
                    "silent": False,
                    # "summary_method": "reflection_with_llm",
                },
                {
                    "chat_id": 2,
                    "recipient": self.pim_wcs_searchbot,
                    "message": {"content": str(self.message_history[-1])},
                    "silent": False,
                    # "summary_method": "reflection_with_llm",
                },
                {
                    "chat_id": 3,
                    "recipient": self.crossref_searchbot,
                    "message": {"content": str(self.message_history[-1])},
                    "silent": False,
                },
                # {
                #     "chat_id": 4,
                #     "prerequisites": [1, 2, 3],
                #     "recipient": self.user_proxy,
                #     "silent": False,
                #     "message": "",
                # },
            ]
        )

Screenshots and logs

Additional Information

No response

[sonichi]

Sounds that you need agents like described in #2223
Let's see who will implement that first.