[FR]: Add a field to ImagenInlineImage for the base64 encoded image

[thatfiredev]

Description

TL;DR: Should we add an extra field to ImagenInlineImage that exposes the bytesBase64Encoded from the Imagen response? (for feature parity with the other SDKs)

All the other Vertex AI in Firebase SDKs return the raw base64 encoded image:

• Flutter
• JS
• Android (the refdocs are currently not explicit about that, but I opened docs: explicitly call out that ImagenInlineImage#data is base64 encoded firebase-android-sdk#6785 to fix it)

Swift seems to decode the base64 image before returning it as data: Data:

firebase-ios-sdk/FirebaseVertexAI/Sources/Types/Public/Imagen/ImagenInlineImage.swift

Line 63 in b01a8f2

let bytesBase64Encoded = try container.decode(String.self, forKey: .bytesBase64Encoded)

API Proposal

No response

Firebase Product(s)

Vertex AI

[andrewheard]

@thatfiredev Do you know why the Android SDK doesn't decode the base64 for the developer in ImagenImageResponse.toPublicInline()? I didn't realize during the API Proposal that the ByteArray returned in the Kotlin API was going to return base64-encoded bytes (I assumed binary).

On Apple platforms, I don't have a use case in mind for the image data encoded as base64. The user journeys I've considered are:

• Displaying the image in an app's UI
  • The UIImage initializer init(data:) fails if the Data is base64-encoded.
  • I'm guessing this is also true on Android since the asBitmap() helper function decodes the base64-encoded ByteArray to a binary ByteArray.
• Storing the generated images in Cloud Storage for Firebase using the Firebase Storage SDK
  • Firebase Storage should accept whatever data format but base64 is less efficient (larger file sizes) compared to raw binary bytes.
• Writing the generated images to disk for future use
  • Same as with GCS, base64 would work but is less efficient.

In both the GCS and disk cases, devs will likely eventually want to show these in a UI where they'll need to decode the base64 anyway.

We could offer an additional bytesBase64Encoded computed property (returning a String like JS or Data like Flutter), e.g.:

var bytesBase64Encoded: String {
  return data.base64EncodedString()
}

// or

var bytesBase64Encoded: Data {
  return data.base64EncodedData()
}

However, I feel that this adds unnecessary confusion to our API surface (e.g., "Should I read the image data from data or from bytesBase64Encoded?) for the sake of parity, without a use-case in mind. Additionally, Swift developers can simply call base64EncodedString(options:) or base64EncodedData(options:) (i.e., the one-liners shown in the computed properties above) if they have a specific need for it.

Note: We had intentionally decided to diverge on this API surface between Swift/Android and JS because, in JS, the base64 string is useful because HTML img tags support base64 as the data source.

@rachelsaunders @paulb777 Do you feel strongly about adding a bytesBase64Encoded property? If so, would it be reasonable to have someone write an API proposal / review before we commit to one of them?

[andrewheard]

cc: @peterfriese to get your thoughts too.

[google-oss-bot]

Hey @thatfiredev. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[thatfiredev]

After some offline discussion we have decided that exposing the bytesBase64Encoded provides no value in Apple platforms or Android.
We also changed the behavior of the Android SDK to return the decoded value instead: firebase/firebase-android-sdk#6800