Using prompts with weights

[roman-verbit-ai]

Hi,

Thank you for this great project guys!
I am trying to use this to create smooth transitions between a sequence of images which I generate by varying prompt weights.
I have a sequence of images being generated live, by using prompts of this general form:
(Beautiful medieval town on a hilltop, with a river flowing in the valley below)1.0, (winter)0.1
Then I vary the weights with each step to get the next image... I need to do it Live, so pre-generating stuff is not an option.

Now, I wanted to use this project to create smooth transitions between each 2 images I have.
To do that, I want to feed two prompts which contain weights, something like this:

1. Prompt 1: (Beautiful medieval town on a hilltop, with a river flowing in the valley below)1.0, (winter)0.1
2. Prompt 2: (Beautiful medieval town on a hilltop, with a river flowing in the valley below)1.0, (winter)0.3

I tried all of the methods listed here: https://huggingface.co/docs/diffusers/en/using-diffusers/weighted_prompts
However, it doesn't seem to react to it.

I tried to introduce compel into the mix by overriding the get_text_embedding function like this:

class BlendingEngineCompel(BlendingEngine):
    def get_text_embedding(self, prompt):

        print('My embeddings are running!')
        compel = Compel(
                        tokenizer=[self.dh.pipe.tokenizer, self.dh.pipe.tokenizer_2],
                        text_encoder=[self.dh.pipe.text_encoder, self.dh.pipe.text_encoder_2],
                        returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED,
                        requires_pooled=[False, True]
                    )

        # create compel prompt components
        conditioning, pooled = compel(prompt)

        do_classifier_free_guidance = self.guidance_scale > 1 and self.dh.pipe.unet.config.time_cond_proj_dim is None
        text_embeddings = self.dh.pipe.encode_prompt(
            # prompt=prompt,
            # prompt_2=prompt,
            device=self.dh.pipe._execution_device,
            num_images_per_prompt=1,
            do_classifier_free_guidance=do_classifier_free_guidance,
            negative_prompt=self.negative_prompt,
            negative_prompt_2=self.negative_prompt,
            prompt_embeds=conditioning,
            pooled_prompt_embeds=pooled,
            negative_prompt_embeds=None,
            negative_pooled_prompt_embeds=None,
            lora_scale=None,
            clip_skip=None,  # self.pipe._clip_skip,
        )
        return text_embeddings

This still does not work.
The results I see are basically that weights are ignored.

Can you please advice on this?
Is it possible to get weights working with the way you process text embeddings?
If not, could it be possible to feed two pre-generated images and create a set of transition images between them?

Any help would be appreciated.
I would also gladly contribute to the repo here if you guide me towards the right solution!