[Bug]: AzureOpenAI not recognizing as MultiModal

[daniyaniazidev]

Bug Description

According to doocumentation Azure Open AI is multmodal llm
https://docs.llamaindex.ai/en/stable/examples/multi_modal/azure_openai_multi_modal/

But its returning false and hence my query engine is giving me assertion error.

Please check and anybody has idea how to resolve this error

Version

Latest

Steps to Reproduce

Create MultiVectore store from image and text node
Create query engine from index and try passing azure open ai as multi modal llm

Relevant Logs/Tracbacks

[dosubot]

To resolve the issue where AzureOpenAI is not recognized as a MultiModalLLM, ensure the following:

1. Correct Class Usage: Make sure you are using the AzureOpenAIMultiModal class, which is specifically designed for multi-modal capabilities. This class includes the _get_multi_modal_chat_message method that processes both text and image documents ^[1].

2. Proper Initialization: Verify that the AzureOpenAI instance is initialized with the correct parameters. This includes setting the engine, model, azure_endpoint, api_key, and api_version. These parameters are crucial for the model to function correctly as a multi-modal LLM ^[2][3].

3. Configuration Check: Double-check your configuration against the documentation to ensure all necessary steps and prerequisites are met, such as setting up an Azure subscription, accessing the Azure OpenAI Service, and deploying the model correctly ^[2].

4. Instance Verification: Ensure that the instance you are passing to as_query_engine is indeed an instance of AzureOpenAIMultiModal and not just AzureOpenAI. The as_query_engine method uses an assert statement to check if the LLM is an instance of MultiModalLLM ^[4].

By following these steps, you should be able to resolve the assertion error and have your AzureOpenAI instance recognized as a MultiModalLLM.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}