Runtime exception when using OpenVinoExecution provider on FP16 onnx model

[hlevring]

Describe the issue

Python 3.11
openvino==2024.4.0
onnxruntime-openvino==1.20.0

I am getting an error like this on Openvino GPU device (issue only in FP16, FP32 works)

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running SkipSimplifiedLayerNormalization node. Name:'/model/layers.0/post_attention_layernorm/SkipLayerNorm' Status Message: C:\Users\Administrator\Desktop\windows_perf\release_validation_1.20.0\final_build\onnxruntime\include\onnxruntime\core/framework/op_kernel_context.h:42 onnxruntime::OpKernelContext::Input Missing Input: model.layers.0.input_layernorm.weight

I just found this ticket with the exactly same error
microsoft#22704

Supposedly the issue should have been fixed in Microsoft repository with 1.20.1, but no new release from Intel.

To reproduce

I created a short script to test the models below, and find the same results and error
TestFP16.py.zip

(BREAKS) https://huggingface.co/onnx-community/tiny-random-LlamaForCausalLM-ONNX/resolve/main/onnx/model_fp16.onnx

(WORKS) https://huggingface.co/onnx-community/tiny-random-LlamaForCausalLM-ONNX/resolve/main/onnx/model.onnx

Urgency

It seems like a pretty urgent and critical issue.

Platform

Windows

OS Version

Windows 11 24H2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

Execution Provider for ONNXRuntime 5.5 c4fb724

ONNX Runtime API

Python

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

No response

[sfatimar]

Hello I do not think 1.20.1 will fix OpenVINO Issue. As SkipLayerNorm is getting added only in 1.20.1 for NPU. I will check the model in GPU for 1.20.1 with OpenVINO and see if it works.

[hlevring]

Thanks it would be great if I can get this working for both OpenVINO GPU and NPU. The example model is not the model I am actually targeting, but I got identical error so it was an easy example.