Running model from local path

[kjoth]

Hi,

Could you provide an example of how to configure an infinite Docker instance to load a model from a local directory instead of referring to the Hugging Face repository?

When tried to refer to local path of a ONNX model it refere to hugging face repository, not the local path.

Run command:
docker run -it -p $port:$port michaelf34/infinity --model-path onnx_distilroberta --port $port --batch-size 10 --device cpu --engine=optimum

Error :

Repository Not Found for url: https://huggingface.co/onnx_distilroberta/resolve/main/config.json.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.

[michaelfeil]

@kjoth For the local model to work inside a docker container, I would highly recommend to interactive ssh. docker run -it --entrypoint /bin/bash and get it to run first.

-v path/to/model/outside:/model/inside

I would be happy about a PR that showcase that. With ONNX/ --engine OPTIMUM models, It might be more complicated, because there are various safeguard that optimize for not downloading all other models from huggingface (jax, rustformers, torch weights) - so It might not even work. Is the model not on HF at all?

[michaelfeil]

Can you share them with full stacktrace and not as screenshot?

[michaelfeil]

I assume, that there is a max_length=514 in the tokenizer config, but it should be set to 512.

[kjoth]

Hi @michaelfeil , Please fidn the full stacktrace.

docker run -it -p $port:$port michaelf34/infinity --model-name-or-path Xenova/all-distilroberta-v1 --port $port --device cpu --engine=optimum 2024-03-27 08:21:29.909 | WARNING | fastembed.embedding:<module>:7 - DefaultEmbedding, FlagEmbedding, JinaEmbedding are deprecated. Use TextEmbedding instead. INFO: Started server process [1] INFO: Waiting for application startup. INFO 2024-03-27 08:21:30,355 infinity_emb INFO: model=Xenova/all-distilroberta-v1 selected, using engine=optimum and device=cpu` select_model.py:54
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 713/713 [00:00<00:00, 6.91MB/s]
INFO 2024-03-27 08:21:31,299 infinity_emb INFO: Found 3 onnx files: [PosixPath('onnx/model.onnx'), PosixPath('onnx/model_fp16.onnx'), PosixPath('onnx/model_quantized.onnx')] utils_optimum.py:177
INFO 2024-03-27 08:21:31,300 infinity_emb INFO: Using onnx/model_quantized.onnx as the model utils_optimum.py:181
The ONNX file onnx/model_quantized.onnx is not a regular name used in optimum.onnxruntime, the ORTModel might not behave as expected.
model_quantized.onnx: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 82.1M/82.1M [00:13<00:00, 6.04MB/s]
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 542/542 [00:00<00:00, 5.61MB/s]
vocab.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 798k/798k [00:00<00:00, 918kB/s]
merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 23.6MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 2.33MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 280/280 [00:00<00:00, 2.46MB/s]
INFO 2024-03-27 08:21:53,065 infinity_emb INFO: Optimizing model utils_optimum.py:101
/app/.venv/lib/python3.10/site-packages/optimum/onnxruntime/configuration.py:770: FutureWarning: optimize_with_onnxruntime_only will be deprecated soon, use enable_transformers_specific_optimizations instead, enable_transformers_specific_optimizations is set to True.
warnings.warn(
/app/.venv/lib/python3.10/site-packages/optimum/onnxruntime/configuration.py:770: FutureWarning: disable_embed_layer_norm will be deprecated soon, use disable_embed_layer_norm_fusion instead, disable_embed_layer_norm_fusion is set to True.
warnings.warn(
Optimizing model...
2024-03-27 08:21:53.380390104 [W:onnxruntime:, inference_session.cc:1868 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.
WARNING 2024-03-27 08:21:53,531 onnxruntime.transformers.optimizer WARNING: Model producer not matched: Expected "pytorch", Got "onnx.quantize".Please specify correct --model_type parameter. optimizer.py:219
WARNING 2024-03-27 08:21:53,621 onnx_model WARNING: shape inference failed which might impact useless cast node detection. onnx_model.py:616
WARNING 2024-03-27 08:21:53,716 fusion_skiplayernorm WARNING: symbolic shape inference disabled or failed. fusion_skiplayernorm.py:34
WARNING 2024-03-27 08:21:53,796 fusion_skiplayernorm WARNING: symbolic shape inference disabled or failed. fusion_skiplayernorm.py:34
INFO 2024-03-27 08:21:53,828 fusion_base INFO: Fused Reshape: 1 fusion_base.py:77
INFO 2024-03-27 08:21:53,830 onnx_model INFO: Removed 3 nodes onnx_model.py:942
INFO 2024-03-27 08:21:53,912 onnx_model_bert INFO: opset version: 11 onnx_model_bert.py:402
INFO 2024-03-27 08:21:53,922 onnx_model INFO: Sort graphs in topological order onnx_model.py:1126
INFO 2024-03-27 08:21:54,060 onnx_model INFO: Model saved to /root/.cache/huggingface/hub/infinity_onnx/Xenova/all-distilroberta-v1/model_quantized_optimized.onnx onnx_model.py:1134
Configuration saved in /root/.cache/huggingface/hub/infinity_onnx/Xenova/all-distilroberta-v1/ort_config.json
Optimized model saved at: /root/.cache/huggingface/hub/infinity_onnx/Xenova/all-distilroberta-v1 (external data format: False; saved all tensor to one file: True)
The ONNX file model_quantized_optimized.onnx is not a regular name used in optimum.onnxruntime, the ORTModel might not behave as expected.
INFO 2024-03-27 08:21:54,880 infinity_emb INFO: Getting timings for batch_size=64 and avg tokens per sentence=4 select_model.py:77
0.82 ms tokenization
29.06 ms inference
0.13 ms post-processing
30.02 ms total
embeddings/sec: 2132.18
2024-03-27 08:21:54.911750313 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running Gather node. Name:'/embeddings/position_embeddings/Gather' Status Message: indices element out of data bounds, idx=514 must be within the inclusive range [-514,513]
ERROR: Traceback (most recent call last):
File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 677, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 566, in aenter
await self._router.startup()
File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 654, in startup
await handler()
File "/app/infinity_emb/infinity_server.py", line 57, in _startup
app.model = AsyncEmbeddingEngine.from_args(engine_args)
File "/app/infinity_emb/engine.py", line 45, in from_args
engine = cls(**asdict(engine_args), _show_deprecation_warning=False)
File "/app/infinity_emb/engine.py", line 36, in init
self._model, self._min_inference_t, self._max_inference_t = select_model(
File "/app/infinity_emb/inference/select_model.py", line 79, in select_model
loaded_engine.warmup(batch_size=engine_args.batch_size, n_tokens=512)
File "/app/infinity_emb/transformer/abstract.py", line 55, in warmup
return run_warmup(self, inp)
File "/app/infinity_emb/transformer/abstract.py", line 105, in run_warmup
embed = model.encode_core(feat)
File "/app/infinity_emb/transformer/embedder/optimum.py", line 95, in encode_core
outputs = self.model(**onnx_input)
File "/app/.venv/lib/python3.10/site-packages/optimum/modeling_base.py", line 91, in call
return self.forward(*args, **kwargs)
File "/app/.venv/lib/python3.10/site-packages/optimum/onnxruntime/modeling_ort.py", line 996, in forward
outputs = self.model.run(None, onnx_inputs)
File "/app/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'/embeddings/position_embeddings/Gather' Status Message: indices element out of data bounds, idx=514 must be within the inclusive range [-514,513]

ERROR: Application startup failed. Exiting.`

[kjoth]

Updated the config.json "max_position_embeddings": 514 to new value "max_position_embeddings": 512, and it works.