Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] TypeError: 'NoneType' object is not subscriptable #255

Open
yinian-lw opened this issue Mar 10, 2025 · 0 comments
Open

[Bug] TypeError: 'NoneType' object is not subscriptable #255

yinian-lw opened this issue Mar 10, 2025 · 0 comments
Assignees
@rlsu9

Comments

@yinian-lw
Copy link

Environment

4xA100-SXM4-80GB

Describe the bug

dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Connected all rings
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Connected all rings
dc61-p20-t1-n036:211711:216372 [0] NCCL INFO Connected all rings
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Connected all rings
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO Connected all trees
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dc61-p20-t1-n036:211711:216372 [0] NCCL INFO Connected all trees
dc61-p20-t1-n036:211711:216372 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
dc61-p20-t1-n036:211711:216372 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO Connected all trees
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO Connected all trees
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dc61-p20-t1-n036:211713:216374 [2] NCCL INFO ncclCommInitRank comm 0x1546df30 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c8000 commId 0x9540fcf47b717b24 - Init COMPLETE
dc61-p20-t1-n036:211711:216372 [0] NCCL INFO ncclCommInitRank comm 0x156aa3c0 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 46000 commId 0x9540fcf47b717b24 - Init COMPLETE
dc61-p20-t1-n036:211712:216373 [1] NCCL INFO ncclCommInitRank comm 0x15071590 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId 4c000 commId 0x9540fcf47b717b24 - Init COMPLETE
dc61-p20-t1-n036:211714:216375 [3] NCCL INFO ncclCommInitRank comm 0x137647f0 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId cc000 commId 0x9540fcf47b717b24 - Init COMPLETE
Traceback (most recent call last):
File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 663, in
main(args)
File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 368, in main
loss, grad_norm = train_one_step(
^^^^^^^^^^^^^^^
File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 146, in train_one_step
model_pred = transformer(**input_kwargs)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 864, in forward
output = self._fsdp_wrapped_module(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: Traceback (most recent call last):
[rank2]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 663, in
[rank2]: main(args)
[rank2]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 368, in main
[rank2]: loss, grad_norm = train_one_step(
[rank2]: ^^^^^^^^^^^^^^^
[rank2]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 146, in train_one_step
[rank2]: model_pred = transformer(**input_kwargs)[0]
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank2]: return self._call_impl(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank2]: return forward_call(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 864, in forward
[rank2]: output = self._fsdp_wrapped_module(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank2]: return self._call_impl(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank2]: return forward_call(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/models/hunyuan/modules/models.py", line 569, in forward
[rank2]: double_block_args = [img, txt, vec, freqs_cis, text_mask, mask_strategy[index]]
[rank2]: ~~~~~~~~~~~~~^^^^^^^
[rank2]: TypeError: 'NoneType' object is not subscriptable
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/models/hunyuan/modules/models.py", line 569, in forward
double_block_args = [img, txt, vec, freqs_cis, text_mask, mask_strategy[index]]
~~~~~~~~~~~~~^^^^^^^
TypeError: 'NoneType' object is not subscriptable
[rank1]: Traceback (most recent call last):
[rank1]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 663, in
[rank1]: main(args)
[rank1]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 368, in main
[rank1]: loss, grad_norm = train_one_step(
[rank1]: ^^^^^^^^^^^^^^^
[rank1]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 146, in train_one_step
[rank1]: model_pred = transformer(**input_kwargs)[0]
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 864, in forward
[rank1]: output = self._fsdp_wrapped_module(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/models/hunyuan/modules/models.py", line 569, in forward
[rank1]: double_block_args = [img, txt, vec, freqs_cis, text_mask, mask_strategy[index]]
[rank1]: ~~~~~~~~~~~~~^^^^^^^
[rank1]: TypeError: 'NoneType' object is not subscriptable
[rank0]: Traceback (most recent call last):
[rank0]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 663, in
[rank0]: main(args)
[rank0]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 368, in main
[rank0]: loss, grad_norm = train_one_step(
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/train.py", line 146, in train_one_step
[rank0]: model_pred = transformer(**input_kwargs)[0]
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 864, in forward
[rank0]: output = self._fsdp_wrapped_module(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/nlhei-nas/liwen.8459/VideoProj/MyFastVideo/fastvideo/models/hunyuan/modules/models.py", line 569, in forward
[rank0]: double_block_args = [img, txt, vec, freqs_cis, text_mask, mask_strategy[index]]
[rank0]: ~~~~~~~~~~~~~^^^^^^^
[rank0]: TypeError: 'NoneType' object is not subscriptable

Reproduction

# export WANDB_BASE_URL="https://api.wandb.ai"
export WANDB_MODE=offline

torchrun --nnodes 1 --nproc_per_node 4 --master_port 32323 \
    fastvideo/train.py \
    --seed 42 \
    --pretrained_model_name_or_path $MODEL_PATH \
    --dit_model_name_or_path $MODEL_PATH/hunyuan-video-t2v-720p/transformers/mp_rank_00_model_states.pt\
    --model_type "hunyuan" \
    --cache_dir data/.cache \
    --data_json_path data/Image-Vid-Finetune-HunYuan/videos2caption.json \
    --validation_prompt_dir data/Image-Vid-Finetune-HunYuan/validation \
    --gradient_checkpointing \
    --train_batch_size=1 \
    --num_latent_t 32 \
    --sp_size 4 \
    --train_sp_batch_size 1 \
    --dataloader_num_workers 4 \
    --gradient_accumulation_steps=1 \
    --max_train_steps=2000 \
    --learning_rate=1e-5 \
    --mixed_precision=bf16 \
    --checkpointing_steps=200 \
    --validation_steps 100 \
    --validation_sampling_steps 50 \
    --checkpoints_total_limit 3 \
    --allow_tf32 \
    --ema_start_step 0 \
    --cfg 0.0 \
    --ema_decay 0.999 \
    --log_validation \
    --output_dir=data/outputs/HSH-Taylor-Finetune-Hunyuan \
    --tracker_project_name HSH-Taylor-Finetune-Hunyuan \
    --num_frames 93 \
    --num_height 720 \
    --num_width 1280 \
    --shift 7 \
    --validation_guidance_scale "1.0"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rlsu9 rlsu9

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yinian-lw @rlsu9