VertexAi Codestral autocompletion response undefined

[eqinox76]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Ubuntu 24.02 (WSL on win 11)
- Continue version: 0.8.66
- IDE version: vscode 1.96.2
- Model: Codestral
- config.json:
  
  "tabAutocompleteModel": {
    "title": "Codestral (Vertex AI)",
    "provider": "vertexai",
    "model": "codestral",
    "projectId": "project",
    "region": "europe-west4"
  },

Description

Setup is working but it seems the autocomplete answer from gcp is mangled.

To reproduce

Use autocomplete with the given settings.

Log output

##### Completion options #####
{
  "contextLength": 32768,
  "model": "codestral",
  "maxTokens": 4096,
  "temperature": 0.01,
  "stop": [
    "[PREFIX]",
    "[SUFFIX]",
    "/src/",
    "#- coding: utf-8",
    ""
  ],
  "raw": true
}

##### Prompt #####
[SUFFIX]
[PREFIX]diff --git a/.gitignore b/.gitignore
index c724ece..e1d9af3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,3 +16,5 @@ gin-bin
 *.out
 
+
+.codegpt
\ No newline at end of file

==========================================================================
==========================================================================
Completion:
undefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefined

[saraangelmurphy]

I am also experiencing the same issue on MacOS. Codestral works as expected when invoked using the MistralGoogleCloud python library and running code locally, or within a Colab runtime environment within GCP. Other VertexAI models work as expected. I do not see anything interesting in the GCP Project Audit logs for either successful calls to Gemini VertexAI models or unsuccessful calls to codestral.

This issue occurs in both the release and prerelease 0.8.66 versions of continue

To Reproduce

• Create a vertex AI project in GCP
• Navigate to "Model Garden" and search for "Codestral"
• Enable the Mistral Codestral Model via the GCP Marketplace (usage based pricing)
• Authenticate to GCP locally using gcloud auth application-default login
• Confirm the model works and is functional using the provided colab notebooks, or with local testing
• Confirm other models like Gemini 1.5 flash/pro work as expected
• Add Codestral as either a chat model, or as the tabAutoComplete model by manually editing the continue config.json file
• Attempt to query codestral either via the chatUI or autocompletions and observed a repeated "undefined" output.

Relevant environment info

- OS: MacOS Sonoma Version 14.7.2 (23H311)
- Continue version: 0.8.66
- IDE version: vscode  1.96.2
- Model: Codestral
- config.json:
  
    {
      "title": "Codestral (Vertex AI)",
      "provider": "vertexai",
      "model": "codestral",
      "projectId": "<MYPROJECTID>",
      "region": "us-central1",
    }
  ],
  "tabAutocompleteModel": {
    "title": "Codestral (Vertex AI)",
    "provider": "vertexai",
    "model": "codestral",
    "projectId": "<MYPROJECTID>",
    "region": "us-central1"
  },
  "tabAutocompleteOptions": {
    "disable": false,
    "debounceDelay": 500,
    "maxPromptTokens": 1500,
    "disableInFiles": ["*.md"]
  },

Log Output:

==========================================================================
==========================================================================
##### Completion options #####
{
  "contextLength": 32768,
  "model": "codestral",
  "maxTokens": 4096,
  "temperature": 0.01,
  "stop": [
    "[PREFIX]",
    "[SUFFIX]",
    "/src/",
    "#- coding: utf-8",
    "```"
  ],
  "raw": true
}

##### Prompt #####
[SUFFIX]
[PREFIX]+++++ Untitled.txt
##### Prompt #####
[SUFFIX]


+++++ Untitled-1
Hello World, my name is foo. Your name is==========================================================================
==========================================================================
##### Completion options #####
{
  "contextLength": 32768,
  "model": "codestral",
  "maxTokens": 4096,
  "temperature": 0.01,
  "stop": [
    "[PREFIX]",
    "[SUFFIX]",
    "/src/",
    "#- coding: utf-8",
    "```"
  ],
  "raw": true
}

##### Prompt #####
[SUFFIX]
[PREFIX]+++++ Untitled.txt
##### Prompt #####
[SUFFIX]


+++++ Untitled-1
Hello World, my name is foo. Your name is ==========================================================================
==========================================================================
Completion:


==========================================================================
==========================================================================
Completion:
undefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefined

[skytern]

Same error:
Jetbrains 0.0.83

"tabAutocompleteModel": {

    "title": "Codestral (Vertex AI)",
    "provider": "vertexai",
    "model": "codestral",
    "projectId": "project_id",
    "region": "us-central1"
  },
  "tabAutocompleteOptions": {
    "debounceDelay": 500,
    "disableInFiles": ["*.md"],
    "maxPromptTokens": 400,
    "prefixPercentage": 0.5,
    "multilineCompletions": "always"
  },

##### Completion options #####
{
  "contextLength": 32768,
  "model": "codestral",
  "maxTokens": 4096,
  "temperature": 0.01,
  "stop": [
    "[PREFIX]",
    "[SUFFIX]",
    "/src/",
    "#- coding: utf-8",
    "```"
  ],
  "raw": true
}

##### Prompt #####
[SUFFIX]
[...]
def func_qui_dit_bonjour():
    print()
    a=1
    b=2
    tota

Completion:
undefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefinedundefined

[johnson-liang]

Also bumping into this on Continue v0.8.66 on VSCode.

However the issue seems related to #3303 . The discussions may be consolidated into 1 issue?

[falcon-tech]

same here

[oboote]

Same here (windows) Continue Version 0.9.259 on VSCode

[scadenalcrs]

I've performed the following test to figure out where those undefined values are coming from:

curl \
   -X POST \
   -H "Authorization: Bearer $(gcloud auth print-access-token)" \
   -H "Content-Type: application/json" \
   "https://us-central1-aiplatform.googleapis.com/v1/projects/[REDACTED]/locations/us-central1/publishers/mistralai/models/codestral-2501:streamRawPredict" \
   --data '{
     "model": "codestral-2501",
 "max_tokens": 4096,
 "temperature": 0.01,
 "stop": [
     "[PREFIX]",
     "[SUFFIX]",
     "/src/",
     "#- coding: utf-8",
     "```"
 ],
 "stream": true,
 "prompt": "[SUFFIX] ",
 "suffix": "[SUFFIX] 123 ",
 "top_p" : 0.2
 }'

It outputs something like:

data: {"id":"d7e03cc7583846a8b35b482da5263776","object":"chat.completion.chunk","created":1739567251,"model":"codestral-2501","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null,"logprobs":null}]}

data: {"id":"d7e03cc7583846a8b35b482da5263776","object":"chat.completion.chunk","created":1739567251,"model":"codestral-2501","choices":[{"index":0,"delta":{"content":"1"},"finish_reason":null,"logprobs":null}]}

data: {"id":"d7e03cc7583846a8b35b482da5263776","object":"chat.completion.chunk","created":1739567251,"model":"codestral-2501","choices":[{"index":0,"delta":{"content":"2"},"finish_reason":null,"logprobs":null}]}

data: {"id":"d7e03cc7583846a8b35b482da5263776","object":"chat.completion.chunk","created":1739567251,"model":"codestral-2501","choices":[{"index":0,"delta":{"content":"3"},"finish_reason":null,"logprobs":null}]}

data: {"id":"d7e03cc7583846a8b35b482da5263776","object":"chat.completion.chunk","created":1739567251,"model":"codestral-2501","choices":[{"index":0,"delta":{"content":"\n"},"finish_reason":null,"logprobs":null}]}

data: {"id":"d7e03cc7583846a8b35b482da5263776","object":"chat.completion.chunk","created":1739567251,"model":"codestral-2501","choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop","logprobs":null}],"usage":{"prompt_tokens":20,"total_tokens":24,"completion_tokens":4}}

event: vertex_event
data: {"usage":{"output_tokens":0}}

data: [DONE]

"event: vertex_event" is properly handled in

continue/core/llm/stream.ts

Line 87 in af175c6

if (data) {

and "data: {"usage":{"output_tokens":0}}" is correctly interpreted in:

continue/core/llm/stream.ts

Line 43 in af175c6

function parseDataLine(line: string): any {

but protected async *StreamFimMistral(..) assumes all chunks have a choices[0].delta.content:

continue/core/llm/llms/VertexAI.ts

Line 340 in af175c6

yield chunk.choices[0].delta.content;

So undefined is yielded before the stream is marked as "DONE".

[Lash-L]

I've added a fix for this in my PR here: #4016