[Bug]: astream chat with tools returns empty response

[alireza-sheikh]

Bug Description

When I ran this code I expected to get the response chunk by chunk but the response was empty. I use the achat_with_tools method and it works properly. (I used gpt-4o and gpt-40-mini)

self.oracle: FunctionCallingLLM

response = await self.oracle.astream_chat_with_tools(
            tools=list(self.toolset.values()),
            chat_history=chat_history,
            tool_choice="required",
        )
async for chunk in response:
     print(chunk.delta)

Version

0.12.10

Steps to Reproduce

I've made my workflow agent with the below guideline.
https://docs.llamaindex.ai/en/stable/examples/workflow/function_calling_agent/

Relevant Logs/Tracbacks

[dosubot]

Hey @alireza-sheikh! I'm here to help you with any issues you're facing. I can assist with bug fixes, answer questions, and guide you on how to contribute. Let's tackle this together!

The issue of astream_chat_with_tools returning an empty response might be related to the configuration of the MockLLM instance. If the MockLLM is initialized with an empty list of responses, it will result in an empty response from the method. This behavior is demonstrated in the empty_calculator_agent fixture, where a MockLLM with no responses leads to an empty output ^[1].

To address this, ensure that the MockLLM or the LLM you are using is properly configured with responses. Additionally, verify that the tools and chat_history are correctly set up and that the tool_choice parameter is appropriately configured. If the issue persists, consider checking the specific implementation details of the FunctionCallingLLM and the tools being used.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[alireza-sheikh]

@dosu.
This is how I create oracle variable and it is using OpenAI object directly and I think it has not any issue.

from llama_index.llms.openai import OpenAI

class LlamaLLMFactory:
    """Factory class to create and configure LLM instances for different tasks."""

    def __init__(self):
        self._fn_cache = {}
        self._cache = {}

    def get_function_calling_llm(
        self, name: str, **kwargs: dict[str, Any]
    ) -> FunctionCallingLLM:
        """Return the requested function calling LLM instance.

        Args:
            name (str): The name of the LLM.
            **kwargs (dict[str, Any]): Additional keyword arguments.

        Returns:
            FunctionCallingLLM: The requested LLM instance.

        Raises:
            ValueError: If the requested LLM is not supported.

        """
        if name in self._fn_cache:
            return self._fn_cache[name]

        if name.startswith("gpt") or name.startswith("o1") or name.startswith("o3"):
            llm_instance = OpenAI(
                api_key=settings.GPT_API_KEY, model=name, additional_kwargs=kwargs
            )
            self._fn_cache[name] = llm_instance
            return llm_instance

        raise ValueError(f"Unsupported LLM: {name}")

llm_factory = LlamaLLMFactory()
oracle = llm_factory.get_function_calling_llm(
                "gpt-3.5-turbo", **llm_args
            )

[logan-markewich]

Can you reproduce in a colab or similar? astream_chat_with_tools works fine for me