Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Vertex AI] Return ImagenInlineImage.data as binary #6800

Merged
merged 7 commits into from
Mar 25, 2025

Conversation

andrewheard
Copy link
Collaborator

@andrewheard andrewheard commented Mar 24, 2025

Updated ImagenGenerationResponse to decode the base64-encoded image data from the backend when creating a public ImagenInlineImage. The ImagenInlineImage.data now returns the raw
image bytes (in JPEG or PNG format, specified in ImagenInlineImage.mimeType), still as a ByteArray, instead of Base64-encoded data. See firebase/firebase-ios-sdk#14603 (comment) for the iOS-equivalent discussion, as well as #6786. This makes Android implementation consistent with the Swift SDK.

Copy link
Contributor

github-actions bot commented Mar 24, 2025

Javadoc Changes:
--- /home/runner/diff/original/firebase-kotlindoc/android/com/google/firebase/vertexai/type/ImagenInlineImage.html	2025-03-25 17:33:46.710833161 +0000
+++ /home/runner/diff/modified/firebase-kotlindoc/android/com/google/firebase/vertexai/type/ImagenInlineImage.html	2025-03-25 17:31:37.539602546 +0000
@@ -11,7 +11,7 @@
       <pre>@<a href="/docs/reference/android/com/google/firebase/vertexai/type/PublicPreviewAPI.html">PublicPreviewAPI</a><br>public final class <a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html">ImagenInlineImage</a></pre>
     </p>
     <hr>
-    <p>Represents an Imagen-generated image that is contained inline</p>
+    <p>Represents an Imagen-generated image that is returned as inline data.</p>
     <h2>Summary</h2>
     <div class="devsite-table-wrapper">
       <table class="responsive">
@@ -29,14 +29,14 @@
             <td><code>final @<a href="https://developer.android.com/reference/kotlin/androidx/annotation/NonNull.html">NonNull</a> byte[]</code></td>
             <td>
               <div><code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html#data()">data</a></code></div>
-              <p>Contains the raw bytes of the image</p>
+              <p>The raw image bytes in JPEG or PNG format, as specified by <code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></code>.</p>
             </td>
           </tr>
           <tr>
             <td><code>final @<a href="https://developer.android.com/reference/kotlin/androidx/annotation/NonNull.html">NonNull</a> <a href="https://developer.android.com/reference/kotlin/java/lang/String.html">String</a></code></td>
             <td>
               <div><code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></code></div>
-              <p>Contains the MIME type of the image (for example, <code>&quot;image/png&quot;</code>)</p>
+              <p>The IANA standard MIME type of the image data; either <code>&quot;image/png&quot;</code> or <code>&quot;image/jpeg&quot;</code>; to request a different format, see <code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenGenerationConfig.html#imageFormat()">ImagenGenerationConfig.imageFormat</a></code>.</p>
             </td>
           </tr>
         </tbody>
@@ -69,12 +69,12 @@
       <div class="api-item"><a name="getData()"></a><a name="setData()"></a><a name="getData--"></a><a name="setData--"></a>
         <h3 class="api-name" id="data()">data</h3>
         <pre class="api-signature no-pretty-print">public&nbsp;final&nbsp;@<a href="https://developer.android.com/reference/kotlin/androidx/annotation/NonNull.html">NonNull</a> byte[]&nbsp;<a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html#data()">data</a></pre>
-        <p>Contains the raw bytes of the image</p>
+        <p>The raw image bytes in JPEG or PNG format, as specified by <code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></code>.</p>
       </div>
       <div class="api-item"><a name="getMimeType()"></a><a name="setMimeType()"></a><a name="getMimeType--"></a><a name="setMimeType--"></a>
         <h3 class="api-name" id="mimeType()">mimeType</h3>
         <pre class="api-signature no-pretty-print">public&nbsp;final&nbsp;@<a href="https://developer.android.com/reference/kotlin/androidx/annotation/NonNull.html">NonNull</a> <a href="https://developer.android.com/reference/kotlin/java/lang/String.html">String</a>&nbsp;<a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></pre>
-        <p>Contains the MIME type of the image (for example, <code>&quot;image/png&quot;</code>)</p>
+        <p>The IANA standard MIME type of the image data; either <code>&quot;image/png&quot;</code> or <code>&quot;image/jpeg&quot;</code>; to request a different format, see <code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenGenerationConfig.html#imageFormat()">ImagenGenerationConfig.imageFormat</a></code>.</p>
       </div>
     </div>
     <div class="list">
--- /home/runner/diff/original/firebase-kotlindoc/android/com/google/firebase/vertexai/type/package-summary.html	2025-03-25 17:33:46.711833163 +0000
+++ /home/runner/diff/modified/firebase-kotlindoc/android/com/google/firebase/vertexai/type/package-summary.html	2025-03-25 17:31:37.541602549 +0000
@@ -227,7 +227,7 @@
           <tr>
             <td><code><a href="/docs/reference/android/com/google/firebase/vertexai/type/ImagenInlineImage.html">ImagenInlineImage</a></code></td>
             <td>
-              <p>Represents an Imagen-generated image that is contained inline</p>
+              <p>Represents an Imagen-generated image that is returned as inline data.</p>
             </td>
           </tr>
           <tr>
--- /home/runner/diff/original/firebase-kotlindoc/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html	2025-03-25 17:33:46.702833147 +0000
+++ /home/runner/diff/modified/firebase-kotlindoc/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html	2025-03-25 17:31:37.529602528 +0000
@@ -11,7 +11,7 @@
       <pre>@<a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/PublicPreviewAPI.html">PublicPreviewAPI</a><br>class <a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html">ImagenInlineImage</a></pre>
     </p>
     <hr>
-    <p>Represents an Imagen-generated image that is contained inline</p>
+    <p>Represents an Imagen-generated image that is returned as inline data.</p>
     <h2>Summary</h2>
     <div class="devsite-table-wrapper">
       <table class="responsive">
@@ -51,14 +51,14 @@
             <td><code><a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-byte-array/index.html">ByteArray</a></code></td>
             <td>
               <div><code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html#data()">data</a></code></div>
-              <p>Contains the raw bytes of the image</p>
+              <p>The raw image bytes in JPEG or PNG format, as specified by <code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></code>.</p>
             </td>
           </tr>
           <tr>
             <td><code><a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-string/index.html">String</a></code></td>
             <td>
               <div><code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></code></div>
-              <p>Contains the MIME type of the image (for example, <code>&quot;image/png&quot;</code>)</p>
+              <p>The IANA standard MIME type of the image data; either <code>&quot;image/png&quot;</code> or <code>&quot;image/jpeg&quot;</code>; to request a different format, see <code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenGenerationConfig.html#imageFormat()">ImagenGenerationConfig.imageFormat</a></code>.</p>
             </td>
           </tr>
         </tbody>
@@ -77,12 +77,12 @@
       <div class="api-item"><a name="getData()"></a><a name="setData()"></a><a name="getData--"></a><a name="setData--"></a>
         <h3 class="api-name" id="data()">data</h3>
         <pre class="api-signature no-pretty-print">val&nbsp;<a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html#data()">data</a>:&nbsp;<a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-byte-array/index.html">ByteArray</a></pre>
-        <p>Contains the raw bytes of the image</p>
+        <p>The raw image bytes in JPEG or PNG format, as specified by <code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a></code>.</p>
       </div>
       <div class="api-item"><a name="getMimeType()"></a><a name="setMimeType()"></a><a name="getMimeType--"></a><a name="setMimeType--"></a>
         <h3 class="api-name" id="mimeType()">mimeType</h3>
         <pre class="api-signature no-pretty-print">val&nbsp;<a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html#mimeType()">mimeType</a>:&nbsp;<a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-string/index.html">String</a></pre>
-        <p>Contains the MIME type of the image (for example, <code>&quot;image/png&quot;</code>)</p>
+        <p>The IANA standard MIME type of the image data; either <code>&quot;image/png&quot;</code> or <code>&quot;image/jpeg&quot;</code>; to request a different format, see <code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenGenerationConfig.html#imageFormat()">ImagenGenerationConfig.imageFormat</a></code>.</p>
       </div>
     </div>
   </body>
--- /home/runner/diff/original/firebase-kotlindoc/kotlin/com/google/firebase/vertexai/type/package-summary.html	2025-03-25 17:33:46.703833148 +0000
+++ /home/runner/diff/modified/firebase-kotlindoc/kotlin/com/google/firebase/vertexai/type/package-summary.html	2025-03-25 17:31:37.530602530 +0000
@@ -200,7 +200,7 @@
           <tr>
             <td><code><a href="/docs/reference/kotlin/com/google/firebase/vertexai/type/ImagenInlineImage.html">ImagenInlineImage</a></code></td>
             <td>
-              <p>Represents an Imagen-generated image that is contained inline</p>
+              <p>Represents an Imagen-generated image that is returned as inline data.</p>
             </td>
           </tr>
           <tr>

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Mar 24, 2025

Coverage Report 1

Affected Products

No changes between base commit (baa335c) and merge commit (02ce2d6).

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/jGi2juMi1x.html

Copy link
Contributor

github-actions bot commented Mar 24, 2025

Test Results

 18 files   -  1 016   18 suites   - 1 016   12s ⏱️ - 34m 19s
 85 tests  -  5 786   85 ✅  -  5 764  0 💤  - 22  0 ❌ ±0 
170 runs   - 11 635  170 ✅  - 11 591  0 💤  - 44  0 ❌ ±0 

Results for commit 362ec20. ± Comparison against base commit baa335c.

This pull request removes 5786 tests.
com.google.android.datatransport.cct.CctBackendFactoryTest ‑ create_returnCCTBackend_WhenBackendNameIsCCT
com.google.android.datatransport.cct.CctDestinationTest ‑ cctDestination_shouldOnlySupportProtoAndJson
com.google.android.datatransport.cct.CctDestinationTest ‑ cctDestination_shouldSupportProtoAndJson
com.google.android.datatransport.cct.CctTransportBackendTest ‑ decorate_whenOffline_shouldProperlyPopulateNetworkInfo
com.google.android.datatransport.cct.CctTransportBackendTest ‑ decorate_whenOnline_shouldProperlyPopulateNetworkInfo
com.google.android.datatransport.cct.CctTransportBackendTest ‑ schedule_shouldAddCookieOnPseudonymousIds
com.google.android.datatransport.cct.CctTransportBackendTest ‑ schedule_shouldDropCookieOnMixedPseudonymousIds
com.google.android.datatransport.cct.CctTransportBackendTest ‑ send_CompressedResponseIsUncompressed
com.google.android.datatransport.cct.CctTransportBackendTest ‑ send_whenBackendRedirectsMoreThan5Times_shouldOnlyRedirect4Times
com.google.android.datatransport.cct.CctTransportBackendTest ‑ send_whenBackendRedirects_shouldCorrectlyFollowTheRedirectViaPost
…

♻️ This comment has been updated with latest results.

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Mar 24, 2025

Size Report 1

Affected Products

  • firebase-vertexai

    TypeBase (baa335c)Merge (02ce2d6)Diff
    aar579 kB579 kB-101 B (-0.0%)
    apk (aggressive)1.40 MB1.40 MB+28 B (+0.0%)
    apk (release)9.37 MB9.37 MB-4 B (-0.0%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/3XmUPGBmmT.html

@google-oss-bot
Copy link
Contributor

The public api surface has changed for the subproject firebase-dataconnect_api.txt:

Please update the api.txt files for the subprojects being affected by this change by running ./gradlew ${subproject}:generateApiTxtFile. Also perform a major/minor bump accordingly.

Copy link
Collaborator

@rlazo rlazo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've manually tested the changes and they work as intended.

Copy link
Member

@thatfiredev thatfiredev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@google-oss-bot
Copy link
Contributor

The public api surface has changed for the subproject firebase-dataconnect_api.txt:

Please update the api.txt files for the subprojects being affected by this change by running ./gradlew ${subproject}:generateApiTxtFile. Also perform a major/minor bump accordingly.

@andrewheard andrewheard merged commit 564734a into main Mar 25, 2025
30 checks passed
@andrewheard andrewheard deleted the ah/imagen-base64-data branch March 25, 2025 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants