[python] AgentGroupChat tool-bleed between agents

[drdrewAQ]

I have an implementation of two agents, each with their own kernel plugin, together in an AgentGroupChat. I've noticed that, on occasion, the agents get confused about which tools are available. AgentTwo will attempt to call tools from AgentOnePlugin, and vice-versa.

The ChatHistory seems to be the culprit, AFAICT, of this information leak between agents. The xml output includes <function_call name="AgentOnePluginName-agent_one_tool">{args}, and similarly the same 'name' appears in the <function_result> tag. It seems that AgentTwo sometimes acts on this 'name' in the history and attempts to call it.

Overriding the __str__ and/or to_prompt methods on ChatHistory seems like one way to proceed, though causing downstream issues with re-instantiation via from_rendered_prompt (which isn't relevant for me at the moment, but I'm doing all of this to evaluate viability for a much larger project -- and this could easily be an issue). There's also the option of trying to prompt-engineer my way out of this on each agent's instructions, but that's clearly a fingers-crossed approach.

Any advice on the 'right' way to handle this?

[moonbox3]

Hi @drdrewAQ thanks for bringing this up. For our understanding, are you using two chat completion agents or a chat completion agent and an assistant agent? My guess is you have two chat completion agents, but I'd like to be sure.

Also, what model are you using? An agent (and thus LLM) should be paying attention to the tools it was provided -- I haven't seen a model try to call tools based on previous context provided in the messages dict. Per your case, that is indeed what is happening, so that is very interesting.

Let me investigate a bit and I'll get back to you ASAP.

[drdrewAQ]

On the first part, you are correct -- this is indeed two chat completion agents.

For the model, I have been using Claude 3.5 Sonnet (Oct 2024).

After further testing, I have now seen instances where AgentTwo is calling tools from the AgentOnePlugin before they've been seen in the chat history. See the attached ChatHistory:

• L296 CroupierAgent references 'Player-place_bet' (which has not appeared in the ChatHistory yet)
• L494-551 CroupierAgent attempts to call 'Player-place_bet' (arguments get knackered, presumably due in part to the absence of the tool)

chat_history-complete-1737936835.json

[moonbox3]

@drdrewAQ Thanks for your response. To confirm: are you using a unique kernel instance (and tool definitions) for each agent? We do see differing levels of performance between models - this is expected as not all models work exactly the same.

Outside of agents, do you see any issues with running Sonnet with normal chat completion operations?

[drdrewAQ]

Indeed, each agent receives a newly instantiated kernel with their respective plugin added.

def _create_kernel_with_chat_completion(service_id: str) -> Kernel:
    kernel = Kernel()
    chat_completion = OpenAIChatCompletion(
        ai_model_id="bedrock-claude-3-5-sonnet-v2",
        api_key=LITELLM_API_KEY,
        service_id=service_id,
    )
    chat_completion.client.base_url = LITELLM_BASE_URL
    kernel.add_service(chat_completion)
    return kernel

def _create_kernel_with_player_tools(service_id: str, player: Player) -> Kernel:
    kernel = _create_kernel_with_chat_completion(service_id)
    kernel.add_plugin(
        PlayerPlugin(player),
        plugin_name="Player",
    )
    return kernel

def _create_kernel_with_croupier_tools(service_id: str, croupier: Croupier) -> Kernel:
    kernel = _create_kernel_with_chat_completion(service_id)
    kernel.add_plugin(
        CroupierPlugin(croupier),
        plugin_name="Croupier",
    )
    return kernel

As far as performance goes, Sonnet has been outstanding with its handling of tool calls and normal chat completion.

[moonbox3]

Thanks for confirming, @drdrewAQ. For this model type, it may be worth exploring how you might manage the chat history that the agents use. Since every ChatMessageContent (and its derivative types) are shared between agents in a group chat right now, you may need to remove content types like FunctionCallContent and FunctionResultContent so that information doesn't bleed to another agent. This would involve getting the messages from the chat, filtering them, clearing the chat, and then adding the messages back.

[drdrewAQ]

I was actually in the process of implementing this approach -- reviewing/filtering/rebuilding the ChatHistory -- when I noticed something strange about one of my log files, which I referenced above:

After further testing, I have now seen instances where AgentTwo is calling tools from the AgentOnePlugin before they've been seen in the chat history.

The complete (unfiltered) chat history attached to that message does not show any evidence of exposing the PlayerPlugin tool (Player-place_bet) to the CroupierAgent via the chat history. Yet the second agent calls this tool from the first agent's plugin (with proper structure for the arguments -- the tool_call is valid except for being called by an agent without 'access' to the plugin/function).

This would seem to be evidence that my initial suspicion about the ChatHistory being the source of the 'leak' was misplaced.