SelectorGroupChat leveraging Ollama model does not select speaker

[brettbourgeois]

Hey folks, I've been experimenting with autogen 0.4.5 and running into issues leveraging the SelectorGroupChat with ollama. I was looking for assistance first before opening an issue.

I'm attempting to run examples from documentation: https://microsoft.github.io/autogen/stable//user-guide/agentchat-user-guide/selector-group-chat.html

Speaker selection fails to select a speaker

File "/home/brett/.local/lib/python3.10/site-packages/autogen_agentchat/teams/_group_chat/_selector_group_chat.py", line 139, in select_speaker
    raise ValueError(f"Expected exactly one agent to be mentioned, but got {mentions}")
ValueError: Expected exactly one agent to be mentioned, but got {}

For environment, everything is running everything locally on docker containers. I'm leveraging ollama/ollama:latest to serve various models. I've tried a few different models (llama3:8b, llama3.2:3b, llama3.3:70b) with the same result.

Here is my OpenAIChatCompletionChat for reference

OpenAIChatCompletionClient(
    model="llama3.3:70b",
    base_url="http://localhost:8443/v1",
    api_key="placeholder",
    model_info={
        "vision": False,
        "function_calling": True,
        "json_output": False,
        "family": "unknown",
    },
)

Troubleshooting

• Added selector_func to force initial speaker (the first speaker is selected but I get the same error when it returns None to defer to the model)

def selector_func(messages: Sequence[AgentEvent | ChatMessage]) -> str | None:
    if messages[-1].source != planning_agent.name:
        return planning_agent.name
    return None

• Added verbose ollama server logging which shows the pre-prompt to select a speaker but I don't think it's returning anything:

ollama-backend-1  | time=2025-02-05T20:29:53.423Z level=DEBUG source=routes.go:1470 msg="chat request" images=0 prompt="<|start_header_id|>system<|end_header_id|>\n\nYou are in a role play game. The following roles are available:\nPlanningAgent: An agent for planning tasks, this agent should be the first to engage when given a new task.\nWebSearchAgent: A web search agent.\nDataAnalystAgent: A data analyst agent. Useful for performing calculations..\nRead the following conversation. Then select the next role from ['PlanningAgent', 'WebSearchAgent', 'DataAnalystAgent'] to play. Only return the role.\n\nuser: Who was the Miami Heat player with the highest points in the 2006-2007 season, and what was the percentage change in his total rebounds between the 2007-2008 and 2008-2009 seasons?\nPlanningAgent: I'll divide this task into two parts:\n1. Find out which Miami Heat player had the highest points in the 2006-2007 season.\n\n    * Web search agent: Search for information on \"Miami Heat players 2006-2007\" \n\n2. Calculate the percentage change in rebounds between the 2007-2008 and 2008-2009 seasons for that player.\n\n    * Data analyst: Perform calculations on rebound data\n\n<tool_call>\n\nRead the above conversation. Then select the next role from ['PlanningAgent', 'WebSearchAgent', 'DataAnalystAgent'] to play. Only return the role.\n<|eot_id|>"
ollama-backend-1  | time=2025-02-05T20:29:53.424Z level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=254 prompt=281 used=4 remaining=277
ollama-backend-1  | time=2025-02-05T20:29:53.691Z level=DEBUG source=runner.go:502 msg="hit stop token" pending=[<|start_header_id|>] stop=<|start_header_id|>
ollama-backend-1  | [GIN] 2025/02/05 - 20:29:53 | 200 |  279.590451ms |      172.22.0.1 | POST     "/v1/chat/completions"
ollama-backend-1  | time=2025-02-05T20:29:53.692Z level=DEBUG source=sched.go:407 msg="context for request finished"

[ekzhu]

cc @afourney for related interest.

@brettbourgeois have you tried to customize the selector's prompt as part of the SelectorGroupChat's constructor argument? The prompt was designed for GPT-4 way back so it might be a bad choice for Llama models.

Alternatively, just try to call the llama model directly in the selector_func, and try things like, structured output or JSON model.

[brettbourgeois]

I haven't experimented enough with the SelectorGroupChat to confidently say yes. I'll play around with this today and follow up.

[afourney]

Yeah, I've been working with llama, and other smaller models, for the last little while, slowly improving compatibility with built-in agents. I have not yet tackled SelectorGroupChat. Does this happen right away, or after the conversation grows for a bit?

I will look into this tomorrow (hopefully), and see if we can tailor the prompts or context setup better for llama.

[brettbourgeois]

It happens immediately if not provided with a selector_func and anytime None is returned by the selector_func.

[afourney]

Ok, thanks. I will look into it today.

[nickrwann]

Issue Diagnosis

Hey @brettbourgeois, @ekzhu, @afourney,

I believe the issue comes down to how Ollama handles system vs. user messages. Right now, the selector prompt includes the entire conversation history inside the system message, which works fine for GPT-4 but doesn’t seem to work well with Llama-based models.

From what I’ve seen, Llama models need the conversation history explicitly separated as user input rather than being embedded inside system instructions. Otherwise, they don’t return anything useful, leading to mentions = {} being empty.

---

Proposed Fix

Instead of embedding {history} inside the system message, I tested a fix where:

1. Removed {history} from the system message
2. Added it separately as a UserMessage

This keeps the selector prompt as a system instruction while providing the conversation history as structured user input, which seems to work better for Llama models.

Before (doesn’t work)

select_speaker_messages = [
    SystemMessage(content=select_speaker_prompt)  # Instructions to model
]

After (fixes it)

select_speaker_messages = [
    SystemMessage(content=select_speaker_prompt),  # Instructions to the model
    UserMessage(content=history, source="user")   # Conversation history as user input
]

---

Results

With this change, Ollama actually picks a speaker instead of failing silently.

Autogen Client Output:

finish_reason='stop' content='DataAnalystAgent' usage=RequestUsage(prompt_tokens=200, completion_tokens=5) cached=False logprobs=None thought=None

Relevant Ollama Log (showing updated request):

2025-02-06T21:02:22.014144629Z time=2025-02-06T21:02:22.014Z level=DEBUG source=routes.go:1470 msg="chat request" images=0 prompt="<|start_header_id|>system<|end_header_id|>\n\nYou are in a role play game. The following roles are available:\n        PlanningAgent: An agent for planning tasks, this agent should be the first to engage when given a new task.\nWebSearchAgent: A web search agent that retrieves information.\nDataAnalystAgent: A data analyst agent. Useful for performing calculations..\n        Read the following conversation. Then select the next role from ['PlanningAgent', 'WebSearchAgent', 'DataAnalystAgent'] to play. Only return the role.\n        <|eot_id|><|start_header_id|>user<|end_header_id|>\n\nuser: Who was the Miami Heat player with the highest points in the 2006-2007 season, and what was the percentage change in his total rebounds between the 2007-2008 and 2008-2009 seasons?<|eot_id|>"

This confirms that Ollama is now receiving a properly formatted message and returning a valid selection.

[afourney]

@nickrwann I came to the same conclusion, and was working on an independent PR: #5409

Please have a look and let me know if it addresses the issue for you.

[nickrwann]

@afourney Ah just saw this. Code looks good! Thanks for the fix.

[afourney]

Ok, I think this should be fixed. I will close the thread, but please re-open if the patch isn't working for you.

[brettbourgeois]

Thank you sir! Appreciate all the hard work you folks are doing with AutoGen.