hf_T5_generate torch.complie() failed with latest transformers==4.44.2

[yiming0416]

Latest transformers=4.44.2 packages from HuggingFace uses torch.isin() which prevents graph break in certain control flows. It also adds the condition branch to avoid copy.deepcopy which is not supported by dynamo

See this PR: huggingface/transformers#30788

However, when I update to transformers==4.44.2. I got the error when running with the following command.

Repro

python benchmarks/dynamo/torchbench.py --accuracy --inference --bfloat16 --device cuda --only=hf_T5_generate

Error log

loading model: 0it [00:45, ?it/s]
cuda eval  hf_T5_generate                     
ERROR:common:
Traceback (most recent call last):
  File "/home/yimingzhou/pytorch/benchmarks/dynamo/common.py", line 2818, in check_accuracy
    new_result = optimized_model_iter_fn(model_copy, example_inputs)
  File "/home/yimingzhou/pytorch/torch/_dynamo/eval_frame.py", line 465, in _fn
    return fn(*args, **kwargs)
  File "/home/yimingzhou/pytorch/benchmarks/dynamo/common.py", line 2544, in run_n_iterations
    self.model_iter_fn(mod, inputs, collect_outputs=False)
  File "/home/yimingzhou/pytorch/benchmarks/dynamo/torchbench.py", line 435, in forward_pass
    return mod(*inputs)
  File "/home/yimingzhou/pytorch/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/yimingzhou/pytorch/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yimingzhou/torchbenchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 189, in forward
    return self.model.generate(inputs, self.generation_config)
  File "/home/yimingzhou/pytorch/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/yimingzhou/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 1713, in generate
    self._prepare_special_tokens(generation_config, kwargs_has_attention_mask, device=device)
  File "/home/yimingzhou/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 1556, in _prepare_special_tokens
    raise ValueError(
ValueError: `decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.
TorchDynamo optimized model failed to run because of following error
fail_to_run

cc @angelayi @desertfire

[FindHao]

Is this the only model having this issue?

[FindHao]

Can you try to add a decoder_start_token_id=None in here?

benchmark/torchbenchmark/util/framework/huggingface/model_factory.py

Line 162 in e6251f1

max_new_tokens=256,

[FindHao]

upstream issue huggingface/transformers#30892

[yiming0416]

Opened the issue in HuggingFace transformers repo: huggingface/transformers#33283