[Bug]: The new system prompt is almost 6k, too large for GPT-4

[enyst]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Describe the bug and reproduction steps

GPT-4 is unusable after 0.28.1, the app fails immediately with a ContextWindowExceeededError.

The underlying error is:

("litellm.BadRequestError: ContextWindowExceededError: OpenAIException - This model's maximum context length is 8192 tokens. 
However, you requested 9836 tokens (5740 in the messages, 4096 in the completion). 
Please reduce the length of the messages or completion.",)

Ugh.

1. 5740 input tokens at the first request is due in part to the system prompt change. The old prompt was 3-4k.

2. I think the error itself is something that openai and anthropic have done since reasoning models: they enforce now a context limit beforehand, at input_tokens + max_tokens.

The old behavior used to be different: this case was allowed by the API, and the API would simply cut off the response if it became too large for the context window. Now we can't send 4096 max. 🤔

OpenHands Installation

Docker command in README

OpenHands Version

No response

Operating System

None

Logs, Errors, Screenshots, and Additional Context

No response

[enyst]

While the error is different (no tool use), this is related to:

• [Bug]: Tool descriptions too long for OpenAI's models #7167

It might be fixed or alleviated by

• Separate tool descriptions to support models with limited description length #7258

Maybe with a tweak to apply to GPT-4. cc: @ryanhoangt

[mamoodi]

This still happens with the latest releases so still an issue.