add BAAI/bge-small-en-v1.5 Optimization #1634
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| model_output = self.model.run_session(self.session, model_inputs) | ||
| model_output = model_output.last_hidden_state.numpy() | ||
| # select the last hidden state of the first token (i.e., [CLS]) as the sentence embedding. | ||
| return model_output[:, 0, :] |
Check failure
Code scanning / CodeQL
Potentially uninitialized local variable Error
|
|
||
| The precision will drop when Add or Softmax types of op are quantized, so they are not included. | ||
|
|
||
| | Quantized Ops | precision | |
There was a problem hiding this comment.
This is interesting. Would it be possible to measure the latency for each case so we get what the accuracy vs latency tradeoff is?
If the tradeoff is large maybe we can spend some time investigating the cause of the bad accuracy.
There was a problem hiding this comment.
Sure. Change to draft since I am still working on npu part. The pr is created in cpu machine
| "calibrate_method": "MinMax", | ||
| "quant_preprocess": true, | ||
| "prepare_qnn_config": true, | ||
| "op_types_to_quantize": [ |
There was a problem hiding this comment.
I have a dev branch where I introduce an option called op_type_to_exclude which is used to modify op_types_to_quantize and nodes_to_exclude.
Olive/olive/passes/onnx/quantization.py
Line 57 in 6e5c3b4
Looks like it might be useful here too when it gets merged
There was a problem hiding this comment.
otherwise, we need to know all of the op types present in the model.
There was a problem hiding this comment.
currently use append_first_op_types_to_quantize_list with nodes_to_exclude will do this. Will we also update this logic?
Olive/olive/passes/onnx/quantization.py
Line 409 in 043f7e1
There was a problem hiding this comment.
honestly, I am not sure why this option was added and if it is used for anything right now.
Not sure if we will touch this option and related logic but I plan to update the logic to be able to use op_types_to_exclude and nodes_to_exclude. The op_types_to_exclude has been very useful for me when I know I don't want to quantize all nodes for an op.
There was a problem hiding this comment.
also created this PR in ort microsoft/onnxruntime#23779.
Describe your changes
Checklist before requesting a review
lintrunner -a(Optional) Issue link