OpenAI assistant, SK function and code interpeter - c#

[JP-droidic]

I need guidance on our UC.

We have an OpenAI agent with a code interpreter enabled and access to a function that extracts data from the database based on the user's query.

Depending on the SQL run's result, we either return the full output or the first 50 rows, and we save the rest as an xlsx file on the Azure blob.

Imagine the following UC:

Can you create a PowerPoint where each slide will be a door number, showing a list of people (name, job title ) who live there?

LLM generates SQL, and once executed, it returns 3,000 records in the SK function that generates/executes the SQL we pass as parameter assistantId and inside of that function, we upload the XLSX file to the assistant

var assistant = await await OpenAIAssistantAgent.RetrieveAsync(new Kernel(), GetClientProvider(), assistantId);
       await assistant.UploadFileAsync(file, name);

The file gets uploaded, we can see it in OpenAI storage

Once the SK function run is completed, OpenAI calls an assistant code interpreter in order to generate a powerpoint. Once the call is done, the pptx gets generated but it has just 3 random slides - it looks like it wasn't interacting with the file we uploaded.

Do you know how to debug/is it possible to add files to the OpenAi agent as an agent.Definition.CodeInterpreterFileIds is read-only

[moonbox3]

Tagging @crickman for visibility.

[crickman]

Hi @JP-droidic , this is an interesting case. I would note that if you update the code-interpeter file-ids in agent's definition then that will affect all instances of that agent. Since you are retrieving the same agent for use with multiple requests, it may make sense to associate the file at the thread-level.

This may be accomplished using OpenAIThreadCreationOptions:

https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Agents/OpenAI/OpenAIThreadCreationOptions.cs#L16
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Agents/OpenAI/OpenAIAssistantAgent.cs#L162

OpenAIThreadCreationOptions options = new() { CodeInterpreterFileIds =[fileId] };
string threadId = agent.CreateThreadAsync(options);
agent.InvokeAsync(threadId, ...);

This will allow the same agent to process different requests. Please let me know if this is a useful approach.

[JP-droidic]

Thank you so much @crickman, that makes sense. I was looking for an example.

A follow-up question. The example you showed describes creating a new thread in order to assign files, but what if we have back and forward conversation between a user and assistant.

The user asks a question

• we create an instance of an assistant agent and a thread
• LLM (assistant answers the question)
• user uploads a file - in order to keep the thread and conversation history, shouldn't we be able to update ongoing thread and pass on fileId?