Intermittent API Error with Image Generation Tool: 'Unknown parameter' Issue

[ElisMia]

Hello everyone,

I'm encountering a recurring issue when using an image generation tool via our API. Specifically, after a few successful image generation requests, subsequent requests fail, and the assistant stops responding, requiring a restart. Here's a brief overview of the problem:

Issue Summary:

• After multiple successful image generation requests, subsequent calls result in a 400 Bad Request error with the message:
  Unknown parameter: 'tools[3].function.arguments'
• This error causes the assistant to cease responding to further requests, impacting usability significantly.

Error Details:

• API Endpoint: https://api.openai.com/v1/threads/thread_XXXXXXXXXXXXXXXXX/runs

• Error Message: "Unknown parameter: 'tools[3].function.arguments'"

• Error Type: "invalid_request_error"

Current Observations:

• The issue appears to be related to how arguments are passed in the API call, particularly when the tool is invoked repeatedly in a short span.

• The tool works as expected for initial requests but fails after several uses.

• This problem does not occur if the image generation tool is not invoked.

This is what I'm curently use:

if (run.RequiredAction.Type == "submit_tool_outputs")
{
        manualOutputs = await ProcessToolCallsWithManualHandling(assistant,run.RequiredAction.SubmitToolOutputs.ToolCalls);

        // submit the tool outputs
        run = await run.SubmitToolOutputsAsync((IEnumerable<ToolOutput>)manualOutputs);
        // waiting while run in Queued and InProgress
        run = await run.WaitForStatusChangeAsync();
} 

public static async Task<List<ToolOutput>> ProcessToolCallsWithManualHandling(AssistantResponse assistant, IEnumerable<ToolCall> toolCalls)
{
        var toolOutputs = new List<ToolOutput>();
        var automaticTasks = new List<Task<ToolOutput>>();

        foreach (var toolCall in toolCalls)
        {
                if (toolCall.FunctionCall.Name == "custometool")
                {
                        var output = await HandleCallManually(toolCall);
                        if (output != null)
                        {
                                toolOutputs.Add(output);
                        }
                }
                else // Automatically process other calls
                {
                        automaticTasks.Add(assistant.GetToolOutputAsync(toolCall));
                }
        }
            
        // Await all automatic processing tasks and add to outputs
        var automaticOutputs = await Task.WhenAll(automaticTasks);
        toolOutputs.AddRange(automaticOutputs);
        return toolOutputs;
}

[ElisMia]

here is the error which i get sometimes when image generation tool is called few times in a row:

Exception occurred: [POST:400] <color="cyan">https://api.openai.com/v1/threads/thread_XXXXXXXXXXXXXXXXX/runs Failed!
{
  "request": {
    "url": "https://api.openai.com/v1/threads/thread_XXXXXXXXXXXXXXXXX/runs"
  },
  "response": {
    "code": 400,
    "headers": {
      "Date": "Wed, 12 Jun 2024 21:24:17 GMT",
      "Content-Type": "application/json",
      "Content-Length": "201",
      "Connection": "keep-alive",
      "openai-version": "2020-10-01",
      "openai-organization": "user-XXXXXXXXXXXXXXXXX",
      "x-request-id": "req_XXXXXXXXXXXXXXXXX",
      "openai-processing-ms": "7",
      "strict-transport-security": "max-age=15724800; includeSubDomains",
      "CF-Cache-Status": "DYNAMIC",
      "Server": "cloudflare",
      "CF-RAY": "XXXXXXXXXXXXXXXXX-VIE",
      "alt-svc": "h3=\":443\"; ma=86400"
    },
    "data": 201,
    "body": {
      "error": {
        "message": "Unknown parameter: 'tools[3].function.arguments'.",
        "type": "invalid_request_error",
        "param": "tools[3].function.arguments",
        "code": "unknown_parameter"
      }
    },
    "error": "HTTP/1.1 400 Bad Request\n"
  }
}

it's randomly apperas. when i ask assiatnt to generate image it can first 2 tims generate image url and give it with response (so in the text directly:

Content messageContent;
switch(content.Type)
{
    case ContentType.Text:
        // so, response needs to be retrieved here directly from text 
        break;
    case ContentType.ImageFile:
        // this is null
        break;
    case ContentType.ImageUrl:
        // and this is also null for image generation
        break;
}

, it's not in ImageUrl.Url).

and after that if i ask again about image generation it fails and I see the error above. and then if this error shown assiatnt stops working and I need to restart playmode to talk with assistant again. Because it also stops to answer questins and just give same error every request. This happens only when there is image generation tool. If I don't have it, just my custom tools and regular talks with assiatant, everything works fine.

Do you think it's some kind of bug or do i miss something in function implementation/usage?

I'm not sure if it's the reson, but it's possible becasue url is inside of the message, amybe it's somehow wrongly handing it later, when sending next message to assiatnt. I'm still looking into it, so if I find out some more info, I'll notify here.

[ElisMia]

here is the error with which it exits:

It fails on line where I try to create new run, after adding assistant message:
var run = await thread.CreateRunAsync(assistant);
it's not creating run vbecauuse failing, and it gives me the exception, I showe above in the image. I was testing further and see that t starts after first image generation call. So I can normally chat with assiatnt without image generation and it works just fine, but once I 1 time ask assistant to generate image and it geberates it - then this error appears and I'm not able to talk with assistant anymore.

I use just regular code like in the example from docs:

message = await thread.CreateMessageAsync(message);
var run = await thread.CreateRunAsync(assistant);
run = await run.WaitForStatusChangeAsync();

[StephenHodgson]

This exception is strange. What exactly is not implemented?

[StephenHodgson]

@ElisMia thanks for the detailed reporting!

Opened a new bug ticket to track RageAgainstThePixel/OpenAI-DotNet#328