Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AI] Support Gemini/VertexAI non-sse Streaming #10804

Open
npolshakova opened this issue Mar 11, 2025 · 0 comments
Open

[AI] Support Gemini/VertexAI non-sse Streaming #10804

npolshakova opened this issue Mar 11, 2025 · 0 comments
Labels
Type: Enhancement New feature or request

Comments

@npolshakova
Copy link
Contributor

npolshakova commented Mar 11, 2025 •
edited
Loading

kgateway version

main

Is your feature request related to a problem or particular use case? Please describe.

The new AI apis introduced streaming for Gemini and Vertex AI, but currently only use sse streaming by setting alt=sse as a request param. We should also support the default Gemini streaming response which returns json instead of data chunks .

Describe the solution you'd like

The alt=sse data chunk streaming response looks like this:

curl \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"role": "user", "parts":[{"text":"explain yourself mr.ai"}]}]}' \
  -X POST "https://us-central1-aiplatform.googleapis.com/v1/projects/k-gateway/locations/us-central1/publishers/google/models/gemini-1.5-flash-001:streamGenerateContent?alt=sse" -H "Authorization: Bearer $(gcloud auth print-access-token)"
data: {"candidates": [{"content": {"role": "model","parts": [{"text": "I"}]}}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " am a large language model, trained by Google.\n\n**What does that mean?**"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.04272461,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.059326172},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.01586914,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.064453125},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.051757813,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.037353516},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.044677734,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.034179688}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "\n\n* **Large Language Model (LLM):** I am a type of"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.05493164,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.055908203},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.025512695,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.103515625},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.068359375,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.04272461},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.07080078,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.027954102}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " artificial intelligence (AI) that has been trained on a massive amount of text data. This allows me to understand and generate human-like text.\n* **"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.032470703,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03857422},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.014038086,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.060058594},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.05834961,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.046142578},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.048828125,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.024414063}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "Trained by Google:** I was created and trained by Google using their advanced AI technology.\n* **Text Data:** My training data includes a vast collection of books, articles,"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.029785156,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03515625},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.011657715,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.04663086},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.068359375,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.037353516},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.053466797,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.031982422}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " code, and other forms of text.\n\n**What can I do?**\n\nI can perform a wide range of language-based tasks, including:\n\n* **Generating text:** I can write stories, poems, articles, summaries, and more."}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.020629883,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.031982422},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.0078125,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03857422},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.05419922,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.026000977},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.041992188,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.040283203}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "\n* **Translating languages:** I can translate text between different languages.\n* **Answering questions:** I can provide information and answer questions based on my knowledge.\n* **Summarizing text:** I can condense large amounts of text into concise summaries"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.024780273,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.0390625},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.010009766,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.036132813},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.06640625,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03466797},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.037841797,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.0390625}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": ".\n* **Writing different kinds of creative content:** I can write scripts, lyrics, emails, letters, and more.\n\n**What am I not?**\n\n* **A sentient being:** I do not have my own thoughts, feelings, or consciousness.\n* **A human:** I am an AI,"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.030639648,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.051757813},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.010986328,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03466797},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.072753906,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03564453},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.060058594,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.046142578}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " not a person.\n* **An expert in everything:** My knowledge is based on my training data, which may not be complete or up-to-date.\n\n**How do I work?**\n\nI use a complex statistical model called a transformer network to process and generate text. This network allows me to learn"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.030639648,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.053466797},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.010986328,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.033203125},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.06542969,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.037841797},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.05102539,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.05493164}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " patterns and relationships within language, enabling me to communicate and generate human-like text.\n\n**My purpose:**\n\nMy primary purpose is to assist humans with language-related tasks, providing information, generating creative content, and making communication more efficient.\n\n**In short, I am a powerful AI tool that can understand and generate text"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.02722168,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.04663086},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.006286621,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.024047852},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.05834961,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.036865234},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.037841797,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.037841797}]}],"modelVersion": "gemini-1.5-flash-001"}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": ", but I am not a person and I am not sentient.**\n"}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.041992188,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.07080078},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.009155273,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.025512695},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.09814453,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.061767578},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.044677734,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.048095703}]}],"usageMetadata": {"promptTokenCount": 5,"candidatesTokenCount": 412,"totalTokenCount": 417},"modelVersion": "gemini-1.5-flash-001"}

Versus the non-sse request looks like this:

curl \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"role": "user", "parts":[{"text":"explain yourself mr.ai"}]}]}' \
  -X POST "https://us-central1-aiplatform.googleapis.com/v1/projects/kgateway/locations/us-central1/publishers/google/models/gemini-1.5-flash-001:generateContent" -H "Authorization: Bearer $(gcloud auth print-access-token)"
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "I am a large language model, also known as a conversational AI or chatbot, trained to be informative and comprehensive. I am trained on a massive amount of text data, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions. For example, I can provide summaries of factual topics or create stories.\n\nHere are some key things to know about me:\n\n* **I am not a person.** I am a computer program.\n* **I do not have feelings or emotions.** I process information and respond based on the patterns I have learned from the data I was trained on.\n* **I am constantly learning and improving.** As I am exposed to more data, I become more accurate and capable.\n* **I can be used for many purposes.** I can be used to answer questions, generate text, translate languages, write different kinds of creative content, and more.\n\nIt is important to remember that I am a tool. My responses are generated based on the data I have been trained on and may not always be accurate, unbiased, or appropriate. Always use your own judgment and consult with a human expert when necessary. \n"
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.024414063,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.032470703
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.0154418945,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.02368164
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.061035156,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.02331543
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.048828125,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.043945313
        }
      ],
      "citationMetadata": {
        "citations": [
          {
            "startIndex": 5,
            "endIndex": 350,
            "uri": "https://www.futurelearn.com/info/blog/what-is-generative-ai"
          }
        ]
      },
      "avgLogprobs": -0.27680010023948071
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 5,
    "candidatesTokenCount": 241,
    "totalTokenCount": 246
  },
  "modelVersion": "gemini-1.5-flash-001"
}
➜  curl \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"role": "user", "parts":[{"text":"explain yourself mr.ai"}]}]}' \
  -X POST "https://us-central1-aiplatform.googleapis.com/v1/projects/kgateway/locations/us-central1/publishers/google/models/gemini-1.5-flash-001:generateContent" -H "Authorization: Bearer $(gcloud auth print-access-token)"
➜ curl \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"role": "user", "parts":[{"text":"explain yourself mr.ai"}]}]}' \
  -X POST "https://us-central1-aiplatform.googleapis.com/v1/projects/gloo-ee/locations/us-central1/publishers/google/models/gemini-1.5-flash-001:streamGenerateContent -H "Authorization: Bearer $(gcloud auth print-access-token)"
dquote>
➜  curl \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"role": "user", "parts":[{"text":"explain yourself mr.ai"}]}]}' \
  -X POST "https://us-central1-aiplatform.googleapis.com/v1/projects/gloo-ee/locations/us-central1/publishers/google/models/gemini-1.5-flash-001:streamGenerateContent" -H "Authorization: Bearer $(gcloud auth print-access-token)"
[{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "I"
          }
        ]
      }
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " am a large language model, trained by Google.\n\n**What does that mean?**"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.04272461,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.059326172
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.01586914,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.064453125
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.051757813,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.037353516
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.044677734,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.034179688
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "\n\n* **Large language model:** I'm a computer program that can understand"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.03857422,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.052734375
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.016357422,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.061767578
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.048095703,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.04345703
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.040283203,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.021240234
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " and generate human-like text. \n* **Trained by Google:** I was created and learned to do this by Google, using a massive dataset of text and code."
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.029296875,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.036865234
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.01361084,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.051757813
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.061035156,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03955078
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.04736328,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03173828
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "\n\n**Here's a simple analogy:** Imagine a really, really smart parrot that can read millions of books. It learns the patterns of language and how to"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.064453125,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.05102539
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.013244629,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03564453
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.20214844,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.07470703
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.03955078,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.0390625
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " use words to create sentences, stories, and even poems. \n\n**What can I do?**\n\n* **Provide information:** I can answer your questions based on the information I've been trained on.\n* **Generate text"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.04663086,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.045410156
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.012817383,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03173828
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.12109375,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.049560547
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.03515625,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03173828
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": ":** I can write stories, poems, emails, articles, and even code.\n* **Translate languages:** I can translate text from one language to another.\n* **Summarize information:** I can condense large amounts of text into shorter"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.0390625,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.04345703
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.012451172,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.040771484
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.10498047,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.04736328
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.041992188,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.037841797
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " summaries.\n\n**What am I not?**\n\n* **A person:** I don't have feelings or opinions, and I can't think for myself.\n* **A sentient being:** I don't have consciousness or self-awareness. \n* **A replacement for human interaction:** I can"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.07080078,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.056640625
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.017456055,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03564453
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.14550781,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.068359375
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.071777344,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03955078
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " be a helpful tool, but I can't replace the value of real human connection.\n\n**How do I work?**\n\nI use a complex system of algorithms and neural networks to understand and process language. This allows me to predict the next word in a sentence, generate coherent text, and respond to your questions"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.05102539,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.046142578
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.01361084,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.026367188
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.115722656,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.05029297
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.052734375,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03955078
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " in a meaningful way.\n\n**Important to remember:**\n\n* I'm always learning and evolving.\n* My responses are based on the information I've been trained on.\n* I'm not perfect, and I can sometimes make mistakes.\n* I'm a tool, and it's important"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.036865234,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.0390625
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.012817383,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.021972656
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.08154297,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03466797
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.046142578,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03466797
        }
      ]
    }
  ],
  "modelVersion": "gemini-1.5-flash-001"
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " to use me responsibly.\n\nI'm excited to help you with your questions and requests! 😊 \n"
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.037841797,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03515625
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.014953613,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.024047852
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.075683594,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.03173828
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.057373047,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.0390625
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 5,
    "candidatesTokenCount": 413,
    "totalTokenCount": 418
  },
  "modelVersion": "gemini-1.5-flash-001"
}
]

Describe alternatives you've considered

No response

Additional Context

No response

┆Issue is synchronized with this Asana task by Unito

Describe the solution you'd like

No response
@npolshakova npolshakova added the Type: Enhancement New feature or request label Mar 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@npolshakova