[LoRA] Support Wan #10943

a-r-r-o-w · 2025-03-03T06:55:23Z

Dummy model: https://huggingface.co/finetrainers/Wan2.1-T2V-1.3B-crush-smol-v0

cc @yiyixuxu for pipeline related changes

HuggingFaceDocBuilderDev · 2025-03-03T07:03:09Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

a-r-r-o-w · 2025-03-03T09:32:27Z

Image-to-Video works as expected after the changes to the pipeline. Most of the changes are to address consistency across implementations and fixes to default values.

import torch
from diffusers import AutoencoderKLWan, WanImageToVideoPipeline, UniPCMultistepScheduler
from diffusers.utils import export_to_video, load_image

# Available models: Wan-AI/Wan2.1-I2V-14B-480P, Wan-AI/Wan2.1-I2V-1.3B-720P
model_id = "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers"
vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
scheduler = UniPCMultistepScheduler(prediction_type="flow_prediction", use_flow_sigmas=True, flow_shift=6.0)
pipe = WanImageToVideoPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)
pipe.scheduler = scheduler
pipe.to("cuda")

# height, width = 480, 832
height, width = 480, 704
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
prompt = (
    "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in "
    "the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."
)
negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"

output = pipe(
    image=image,
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=height,
    width=width,
    num_frames=81,
    num_inference_steps=30,
    guidance_scale=3.0
).frames[0]
export_to_video(output, "output2.mp4", fps=15)

output2.mp4

yiyixuxu · 2025-03-04T05:23:37Z

src/diffusers/models/transformers/transformer_wan.py

@@ -114,9 +115,9 @@ def __init__(self, in_features: int, out_features: int):
        self.norm2 = nn.LayerNorm(out_features)

    def forward(self, encoder_hidden_states_image: torch.Tensor) -> torch.Tensor:
-        hidden_states = self.norm1(encoder_hidden_states_image)
+        hidden_states = self.norm1(encoder_hidden_states_image.float()).type_as(encoder_hidden_states_image)


we cannot always assume self.norm1 is always in float32 here. i.e for layers in _keep_in_fp32_modules, user can still cast it into a different dtype if they do something like model.to(torch.float16)

maybe change to FP32LayerNorm ~

sayakpaul · 2025-03-04T08:22:39Z

src/diffusers/pipelines/wan/pipeline_wan.py

-        num_channels_latents: 16,
-        height: int = 720,
-        width: int = 1280,
-        num_latent_frames: int = 21,
+        num_channels_latents: int = 16,
+        height: int = 480,
+        width: int = 832,
+        num_frames: int = 81,


I will trust that these are safe enough changes for this PR.

Yes, should be safe imo. These are the defaults used for inference and training, and the existing values were from copying Hunyuan pipeline I think.

sayakpaul · 2025-03-04T08:23:01Z

src/diffusers/pipelines/wan/pipeline_wan.py

-            height (`int`, defaults to `720`):
+            height (`int`, defaults to `480`):
                The height in pixels of the generated image.
-            width (`int`, defaults to `1280`):
+            width (`int`, defaults to `832`):
                The width in pixels of the generated image.
-            num_frames (`int`, defaults to `129`):
+            num_frames (`int`, defaults to `81`):


Okay the above changes in prepare_latents() make sense to me.

sayakpaul · 2025-03-04T08:25:42Z

tests/lora/utils.py

+                    tower_name = (
+                        "transformer_blocks"
+                        if any(name == "transformer_blocks" for name in named_modules)
+                        else "blocks"
+                    )
+                    transformer_tower = getattr(pipe.transformer, tower_name)
                    has_attn1 = any("attn1" in name for name in named_modules)
                    if has_attn1:
-                        pipe.transformer.transformer_blocks[0].attn1.to_q.lora_A["adapter-1"].weight += float("inf")
+                        transformer_tower[0].attn1.to_q.lora_A["adapter-1"].weight += float("inf")
                    else:
-                        pipe.transformer.transformer_blocks[0].attn.to_q.lora_A["adapter-1"].weight += float("inf")
+                        transformer_tower[0].attn.to_q.lora_A["adapter-1"].weight += float("inf")


Okay for me. Will remove the test here there:

diffusers/tests/lora/test_lora_layers_lumina2.py

Line 142 in 30cef6b

def test_lora_fuse_nan(self):

(I will take care of it after this PR is merged)

sayakpaul

Results look juicy! Thanks for working on this!

a-r-r-o-w · 2025-03-04T16:39:06Z

Another example on this dataset: https://huggingface.co/datasets/finetrainers/3dgs-dissolve!

output.mp4

This is the 1.3B T2V model. Can't yet train the 14B T2V/I2V model because of how slow it would be so going to try and find time for the flash attention backend and context parallel in trainer this weekend!

sayakpaul · 2025-03-04T16:43:39Z

Fantastic! Time for an aesthetic FT run

nitinmukesh · 2025-03-04T16:48:30Z

@a-r-r-o-w

Looks amazing, are you planning to release the lora for us. :)

sayakpaul · 2025-03-04T16:49:22Z

Will be made available through https://huggingface.co/finetrainers.

yiyixuxu

thanks!

a-r-r-o-w · 2025-03-04T19:56:47Z

@nitinmukesh Ofcourse! Here you go: https://huggingface.co/finetrainers/Wan2.1-T2V-1.3B-3dgs-v0

It's a bit overtrained so needs lower lora strength to work. More amazing things on the way with the learnings in mind!

nitinmukesh · 2025-03-05T15:20:18Z

Awesome. Thank you.

update

87b4b9e

a-r-r-o-w added 2 commits March 3, 2025 08:36

refactor image-to-video pipeline

20d738c

update

c089372

fix copied from

ea1352c

a-r-r-o-w added the roadmap Add to current release roadmap label Mar 3, 2025

a-r-r-o-w requested review from yiyixuxu and sayakpaul March 3, 2025 21:08

a-r-r-o-w marked this pull request as ready for review March 3, 2025 21:09

Merge branch 'main' into lora/wan

ca8afba

yiyixuxu reviewed Mar 4, 2025

View reviewed changes

sayakpaul reviewed Mar 4, 2025

View reviewed changes

sayakpaul approved these changes Mar 4, 2025

View reviewed changes

use FP32LayerNorm

40db8aa

yiyixuxu approved these changes Mar 4, 2025

View reviewed changes

a-r-r-o-w merged commit 3ee899f into main Mar 4, 2025
28 of 30 checks passed

a-r-r-o-w deleted the lora/wan branch March 4, 2025 19:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LoRA] Support Wan #10943

[LoRA] Support Wan #10943

a-r-r-o-w commented Mar 3, 2025 •

edited

Loading

HuggingFaceDocBuilderDev commented Mar 3, 2025

a-r-r-o-w commented Mar 3, 2025

yiyixuxu Mar 4, 2025

sayakpaul Mar 4, 2025

a-r-r-o-w Mar 4, 2025

sayakpaul Mar 4, 2025

sayakpaul Mar 4, 2025

sayakpaul left a comment

a-r-r-o-w commented Mar 4, 2025 •

edited

Loading

sayakpaul commented Mar 4, 2025

nitinmukesh commented Mar 4, 2025 •

edited

Loading

sayakpaul commented Mar 4, 2025

yiyixuxu left a comment

a-r-r-o-w commented Mar 4, 2025

nitinmukesh commented Mar 5, 2025

[LoRA] Support Wan #10943

[LoRA] Support Wan #10943

Conversation

a-r-r-o-w commented Mar 3, 2025 • edited Loading

HuggingFaceDocBuilderDev commented Mar 3, 2025

a-r-r-o-w commented Mar 3, 2025

yiyixuxu Mar 4, 2025

Choose a reason for hiding this comment

sayakpaul Mar 4, 2025

Choose a reason for hiding this comment

a-r-r-o-w Mar 4, 2025

Choose a reason for hiding this comment

sayakpaul Mar 4, 2025

Choose a reason for hiding this comment

sayakpaul Mar 4, 2025

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

a-r-r-o-w commented Mar 4, 2025 • edited Loading

sayakpaul commented Mar 4, 2025

nitinmukesh commented Mar 4, 2025 • edited Loading

sayakpaul commented Mar 4, 2025

yiyixuxu left a comment

Choose a reason for hiding this comment

a-r-r-o-w commented Mar 4, 2025

nitinmukesh commented Mar 5, 2025

a-r-r-o-w commented Mar 3, 2025 •

edited

Loading

a-r-r-o-w commented Mar 4, 2025 •

edited

Loading

nitinmukesh commented Mar 4, 2025 •

edited

Loading