Support for Open models through VLLM

[TheMrguiller]

Hi,

I wonder if you support other model providers that could be store locally. Specially, i do not know how it interacts with other services as you use embeddings for the tool.

Thanks

[prasmussen15]

You can use litellm to create an openAI API (from other models) and you should just be able to provide litellm credentials instead of OpenAI credentials. Just make sure that it also has an embedding model configured.

https://www.litellm.ai

[TheMrguiller]

HI @prasmussen15 ,

Thank you so much for your recommendation. I was able to put it to the test, but with no good results. After sending the first completion, the system seems to fail. It has also caused my own LLM server to go down; I host it using VLLM. Is there any approach you can come up with?
The problem happens after sending the following message:

{"messages": [{"role": "system", "content": "You are a helpful assistant that extracts graph edges from provided context."}, {"role": "user", "content": "\n        Given the following context, extract edges (relationships) that need to be added to the knowledge graph:\n        Nodes:\n        []\n\n        \n\n        Episodes:\n        []\n        Kamala Harris is the Attorney General of California. She was previously the district attorney for San Francisco. <-- New Episode\n        \n\n        Extract entity edges based on the content of the current episode, the given nodes, and context from previous episodes.\n\n        Guidelines:\n        1. Create edges only between the provided nodes.\n        2. Each edge should represent a clear relationship between two DISTINCT nodes.\n        3. The relation_type should be a concise, all-caps description of the relationship (e.g., LOVES, IS_FRIENDS_WITH, WORKS_FOR).\n        4. Provide a more detailed fact describing the relationship.\n        5. Consider temporal aspects of relationships when relevant.\n        6. Avoid using the same node as the source and target of a relationship\n\n        Respond with a JSON object in the following format:\n        {\n            \"edges\": [\n                {\n                    \"relation_type\": \"RELATION_TYPE_IN_CAPS\",\n                    \"source_node_uuid\": \"uuid of the source entity node\",\n                    \"target_node_uuid\": \"uuid of the target entity node\",\n                    \"fact\": \"brief description of the relationship\",\n                    \"valid_at\": \"YYYY-MM-DDTHH:MM:SSZ or null if not explicitly mentioned\",\n                    \"invalid_at\": \"YYYY-MM-DDTHH:MM:SSZ or null if ongoing or not explicitly mentioned\"\n                }\n            ]\n        }\n\n        If no edges need to be added, return an empty list for \"edges\".\n        "}], "model": "Qwen2.5", "max_tokens": 4048, "response_format": {"type": "json_object"}, "stream": false, "temperature": 0}