Autogen + Guidance

[nickgannon10]

I'm looking to instantiate some hard guardrails on autogen outputs, and Microsoft's Guidance Repository seems like it has some awesome features. I'm struggling to get the two libraries to play nice.

It seems like both libraries are trying to own the base layer (meaning they are both wrapping the base API call in a way that enables the libraries functionality but hinders the libraries interoperability).

I'm curious if anyone has figured out how to integrate the guidance library into autogen groupchats? I've got some function calling agents that occasionally forget to format their function calls properly and it causes the system to sometimes fail.

Also, if anyone on the autogen or guidance teams at Microsoft sees this. You guys should team up and build the super-library!

[sonichi]

My thoughts: use register_reply to register a custom reply function using guidance, similar to generate_oai_reply

[gagb]

@Harsha-Nori is making some updates to Guidance-- some fix were need to allow this integration.

[nickgannon10]

This is amazing! Thank you guys so much. Also, I'm more than happy to toss in a pull request for an example notebook or two once the updates to Guidance are in. I'm not sure if they will be of any value, but I'll try to be of help in any way I can!

[gagb]

Btw this was added here: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_guidance.ipynb

[nickgannon10]

huge thank you!!!!

[gagb]

@nickgannon10 sharing if you'd like to help with #1309

[ekzhu]

Just to link another relevant discussion. #1202 perhaps we can have a convenient way to create a structured response generator. This seems particularly useful for non OpenAI model and helps them to support function calling. Not sure where that code would live but it can be a huge benefit. Could be designed as something similar to a decorator like what we did for function calling.

[gagb]

I think the notebook example I made will help with that though I haven't test with OSS models yet. Guidance actually works better with those models. Maybe we can create a new issue for that?

[niuhuluzhihao]

I took a brief look at guidance, but my LLM service is provided through the rest interface. API specifications are the same as opneai .How to configure it in guidance?

[gagb]

Can you please describe your issue more? Are you trying to figure out how to connect guidance with your custom LLM api?

[niuhuluzhihao]

Yes. For example, the API of my local LLM is {"model": "Qwen-72B-Chat", "api_key": "NULL", "base_url": "http://10.235.111.10:2003/api/qwen-72b -server/v1"}. This URL complies with the OpenAI specification. How should I access Guidance?