New plugin: AI Image Generator

[nqst]

Initial checklist

• I understand this is a feature request and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Problem

We want to enhance Uppy's functionality by adding AI features. The first candidates are AI image generation and background removal.

Solution

Here are the screenshots of the new AI image generator feature. This includes the updated Image editor (see #5407), which looks much lighter, and adds the "Edit with AI" and "Remove background" features.

1. The Dashboard with the new "AI Image Generator" plugin:

2. Form. It's very similar to the form that we currently have in the Unsplash plugin:

3. Result images. The idea is to generate 2 images by default (this amount is configurable) so the user can select their preferred option. The selection logic is similar to what we have in the Instagram plugin — clicking on an image toggles a bar with the Select button.

4. Updated Image Editor. (Update the Image Editor UI #5407) It contains the new AI features. The "Edit with AI" button toggles a form (see Screenshot 5 below), while the "Remove bg" is a button which immediately triggers the automatic background removal process.

5. Edit with AI mode active.

These screenshots are likely not final, and the mobile version is not ready yet. Comments and suggestions are welcome!

[Murderlon]

@nqst there are currently no loading and error states. What happens when "Edit with AI" or Remove "bg" take 20 seconds? Or fail? Can we add some designs for that?

[nqst]

What happens when "Edit with AI" or Remove "bg" take 20 seconds? Or fail? Can we add some designs for that?

Absolutely! I'll add these.

[Murderlon]

We should also consider splitting this issue into two: image editor changes and the new plugin. They can be worked on separately and are quite different things to build.

[kvz]

Even if https://github.com/transloadit/team-internals/issues/339 isn't finished yet, we could already leverage Replicate.com/Fal.ai as suggested here.

And then use FLUX, while we don't support it yet.

This seemed to be hard, what's the status of this Merlijn? Tim and I would like to push for this

[Murderlon]

By only relying on @uppy/transloadit and the Robot, instead of leveraging third-party APIs, this plugin can now be very simple.

It's 80% done with just 50 lines of code.

Screen.Recording.2024-09-02.at.11.21.20.mov

  search = async () => {
    const transloadit = this.uppy.getPlugin<Transloadit<M, B>>('Transloadit')

    if (!transloadit) {
      throw new Error('ImageGenerator requires the Transloadit plugin')
    }

    this.uppy.on('transloadit:result', (stepName, result) => {
      // TODO: use a deterministic property to get the right result,
      // not stepName which can be changed by the implementer of the template. 
      if (stepName === 'resized') {
        const { results } = this.getPluginState()
        this.setPluginState({ results: [...results, result] })
      }
    })

    try {
      await this.uppy.upload()
    } finally {
      // Normally uppy.upload() is only used to upload your files
      // but here we need it to _get_ the results from the AI image generator Robot.
      // That means users who set `allowMultipleUploadBatches: false` will not
      // be able to actually upload their files, so we reset the state here.
      this.uppy.setState({ allowNewUpload: true })
    }
  }

  private onCheckboxChange = (result: AssemblyResult) => {
    const { checkedResultIds } = this.getPluginState()

    if (checkedResultIds.has(result.id)) {
      checkedResultIds.delete(result.id)
    } else {
      checkedResultIds.add(result.id)
    }

    this.setPluginState({ checkedResultIds })
  }

  private donePicking = () => {
    const { checkedResultIds, results } = this.getPluginState()
    const files = results
      .filter((result) => checkedResultIds.has(result.id))
      .map((result) => ({
        name: result.name,
        data: { size: result.size },
      }))

    this.uppy.addFiles(files)
  }