Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix chat completion image #344

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Sherlock-Holo
Copy link

when marshal the detail field to '"detail": null', it will break vllm openai api server

3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] Error in preprocessing prompt inputs
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] Traceback (most recent call last):
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   File "/data/ai/pyenv/uv/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 181, in create_chat_completion
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     ) = await self._preprocess_chat(
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   File "/data/ai/pyenv/uv/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_engine.py", line 388, in _preprocess_chat
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     conversation, mm_data_future = parse_chat_messages_futures(
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   File "/data/ai/pyenv/uv/vllm/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 951, in parse_chat_messages_futures
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     sub_messages = _parse_chat_message_content(
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   File "/data/ai/pyenv/uv/vllm/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 879, in _parse_chat_message_content
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     result = _parse_chat_message_content_parts(
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   File "/data/ai/pyenv/uv/vllm/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 781, in _parse_chat_message_content_parts
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     for part in parts:
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]                 ^^^^^
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] pydantic_core._pydantic_core.ValidationError: 5 validation errors for ValidatorIterator
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] 1.typed-dict.text
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   Field required [type=missing, input_value={'type': 'image_url', 'im...Jf//Z', 'detail': None}}, input_type=dict]
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     For further information visit https://errors.pydantic.dev/2.10/v/missing
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] 1.typed-dict.type
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   Input should be 'text' [type=literal_error, input_value='image_url', input_type=str]
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     For further information visit https://errors.pydantic.dev/2.10/v/literal_error
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] 1.typed-dict.image_url.detail
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   Input should be 'auto', 'low' or 'high' [type=literal_error, input_value=None, input_type=NoneType]
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     For further information visit https://errors.pydantic.dev/2.10/v/literal_error
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] 1.typed-dict.input_audio
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   Field required [type=missing, input_value={'type': 'image_url', 'im...Jf//Z', 'detail': None}}, input_type=dict]
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     For further information visit https://errors.pydantic.dev/2.10/v/missing
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197] 1.typed-dict.type
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]   Input should be 'input_audio' [type=literal_error, input_value='image_url', input_type=str]
3月 07 16:43:07 mymachine llama-swap[2425068]: ERROR 03-07 16:43:07 serving_chat.py:197]     For further information visit https://errors.pydantic.dev/2.10/v/literal_error

maybe it is vllm problem, but omit the null field is a good idea

when marshal the detail field to '"detail": null', it will break vllm
openai api server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant