Trying to set up different LLMs on different ports, and it ain't working right.

[phalexo]

Lets say I have a tmux shell #1, and a tmux shell #2.

In #1 I set
export CUDA_VISIBLE_DEVICES=0,1

in #2
export CUDA_VISIBLE_DEVICES-2,3

I run in #1
litellm --model ollama/notus-7b-v1.Q6_k:latest

and in #2
litellm --model ollama/orca-2-13b.Q6_K:latest

In #1 litellm comes back with
http://0.0.0.0.0:8000 interface (allegedly OpenAI API compatible)
and on #2
http://0.0.0.0:##### (some other port)

The question is how exactly a particular agent like "coder" or "manager" or "critic" is associated with the specific model???

There is this file

[
{
"model": "orca-2-13b.Q6_K:latest",
"base_url": "http://localhost:8000",
"api_key": "NULL"
},
{
"model": "notus-7b-v1.Q6_k:latest",
"base_url": "http://localhost:8000",
"api_key": "NULL"
},
{
"model": "llama2",
"base_url": "http://localhost:8000",
"api_key": "NULL"
},
{
"model": "mistral",
"base_url": "http://172.17.0.2:8000",
"api_key": "NULL"
},
]

With ports changed to something else on some of them, but AutoGen seems to ignore these ports and just defaults to 8000

And even if AutoGen respected the port assignment to different models, it is STILL not clear how different agents are associated with specific ports.

Thanks.

[rickyloynd-microsoft]

Just to check, have you been able to run autogen examples successfully as written, without specifying any ports?

[phalexo]

Yes, I can do that. Let me correct that. I can run my example, which I mostly copied from an AutoGen notebook.

[phalexo]

Just to check, have you been able to run autogen examples successfully as written, without specifying any ports? I think I found a clue in one of the issues.

[rickyloynd-microsoft]

And have you been able to run each of these 4 local models successfully by itself without specifying ports?

[phalexo]

Yes, I can do this

ollama run model1

etc.... with one caveat

I had to drop the version for ollama from the current one to 0.1.11.

otherwise I get garbage from GPUs, but runs fine on the host.

[rickyloynd-microsoft]

Have you seen any autogen example that uses multiple ports like you are trying to do?

[phalexo]

Not an example as such, but as I mentioned I see a clue in one of the issues. At least I have an idea what to try.