What to specify for --dynamic-to-fixed-shape-dim-param and --dynamic-to-fixed-shape-dim-value for phi3.5 model?

[fujikosu]

I want to run olive auto-opt command for phi3.5 mini model with --device npu and --provider QNNExecutionProvider options. That requires --dynamic-to-fixed-shape-dim-param and --dynamic-to-fixed-shape-dim-value to be configured as well. But the documentation (https://microsoft.github.io/Olive/reference/cli.html#auto-optimization) doesn't seem to provide enough information about what actually needs to be passed to these two params

--dynamic-to-fixed-shape-dim-param
Symbolic parameter names to use for dynamic to fixed shape pass. Required only when using QNNExecutionProvider.

--dynamic-to-fixed-shape-dim-value
Symbolic parameter values to use for dynamic to fixed shape pass. Required only when using QNNExecutionProvider.

Could you give me any guidance about what needs to be passed for these two for phi3.5?

[xiaoyu-work]

QNN EP doesn't support dynamic shape so you need to make dynamic shape fixed. More details can be found at https://onnxruntime.ai/docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.html. for example you can pass batch_size as dim param and 1 as value. I didn't try it myself so I'm not sure if phi3.5 has more dynamic shapes. If so you can find it yourself by checking onnx model file. Let me know if you still have more questions.

[fujikosu]

@xiaoyu-work

Thank you! I checked phi3.5 onnx model on netron and found these four following parameters are dynamic params

• batch_size
• sequence_length
• past_sequence_length
• total_sequence_length

We can set batch_size to 1 but all the other ones would be updated in every loop of token generation that it needs to be dynamic I think? Any guidance on what we can set for those?