ai.djl.engine.EngineException: Model conversion process failed

## Description

I am currently working on deploying a text embedding model to a SageMaker endpoint:

```python
# Reference:
# - https://github.com/deepjavalibrary/djl-serving/blob/2df533b1753a19e77239ea4fdcf291254d1d7a94/serving/docs/lmi/user_guides/embedding-user-guide.md?plain=1#L65
# - https://github.com/aws-samples/sagemaker-genai-hosting-examples/blob/5e0550489bc0f2d1f9b516552f726a31e30a0bdc/rerankers/rerankers-on-amazon-sagemaker.ipynb

# Assumes SageMaker Python SDK is installed. For example: "pip install sagemaker"
import sagemaker
from sagemaker.djl_inference import DJLModel

# Setup role and sagemaker session
role = sagemaker.get_execution_role() # execution role for the endpoint
session = sagemaker.session.Session() # sagemaker session for interacting with different AWS APIs

# Create the SageMaker Model object.
model_id = "intfloat/multilingual-e5-base"
image_uri = "763104351884.dkr.ecr.us-west-2.amazonaws.com/djl-inference:0.31.0-lmi13.0.0-cu124"
env = {
"SERVING_MIN_WORKERS": "1", # make sure min and max Workers are equals when deploy model on GPU
"SERVING_MAX_WORKERS": "1",
"OPTION_ENGINE": "OnnxRuntime",
"SERVING_BATCH_SIZE": "32",
"SERVING_OPTS": "-Dai.djl.logging.level=debug",
}

model = DJLModel(
model_id=model_id,
role=role,
task="text-embedding",
image_uri=image_uri,
env=env,
)

# Deploy your model to a SageMaker Endpoint and create a Predictor to make inference requests
instance_type = "ml.p3.16xlarge"
endpoint_name = sagemaker.utils.name_from_base("multilingual-e5-base")

predictor = model.deploy(initial_instance_count=1, instance_type=instance_type, endpoint_name=endpoint_name)
```

### Expected Behavior

```log
[INFO ] LmiUtils - Converting model to onnx artifacts
[INFO ] LmiUtils - Converting model to onnx artifacts: [djl-convert, --output-dir, /tmp/.djl.ai/onnx/3df3bb83644cbde8659480d7ff2bf0dcc12e18e4, --output-format, OnnxRuntime, -m, intfloat/multilingual-e5-large-instruct, --optimize, O4, --device, cuda]
[INFO ] LmiUtils - Onnx artifacts built successfully
```

### Error Message

```log
CUDA compat package requires Nvidia driver ⩽550.127.08
Current installed Nvidia driver version is 535.216.01
Setup CUDA compatibility libs path to LD_LIBRARY_PATH
/usr/local/cuda/compat:/usr/local/lib/python3.11/dist-packages/nvidia/cudnn/lib/
[DEBUG] Engine - Registering EngineProvider: TensorRT
[DEBUG] Engine - Registering EngineProvider: PyTorch
[DEBUG] Engine - Registering EngineProvider: Python
[DEBUG] Engine - Registering EngineProvider: MPI
[DEBUG] Engine - Registering EngineProvider: TensorFlow
[DEBUG] Engine - Registering EngineProvider: Rust
[DEBUG] Engine - Found default engine: PyTorch
[INFO ] Ec2Utils - DJL will collect telemetry to help us better understand our users' needs, diagnose issues, and deliver additional features. If you would like to learn more or opt-out please go to: https://docs.djl.ai/master/docs/telemetry.html for more information.
[DEBUG] Engine - Registering EngineProvider: OnnxRuntime
[INFO ] ModelServer - Starting model server ...
[INFO ] ModelServer - Starting djl-serving: 0.31.0 ...
[INFO ] ModelServer -
Model server home: /opt/djl
Current directory: /opt/djl
Temp directory: /tmp
Command line: -Dlog4j.configurationFile=/usr/local/djl-serving-0.31.0/conf/log4j2-plain.xml -Xmx1g -Xms1g -XX:+ExitOnOutOfMemoryError -Dai.djl.util.cuda.fork=true -XX:-UseContainerSupport -Dai.djl.logging.level=debug
Number of CPUs: 64
CUDA version: 124 / 70
Number of GPUs: 8
Max heap size: 1024
Config file: /opt/djl/conf/config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8080
Default job_queue_size: 1000
Default batch_size: 32
Default max_batch_delay: 100
Default max_idle_time: 60
Model Store: /opt/ml/model
Initial Models: ALL
Netty threads: 0
Maximum Request Size: 67108864
Environment variables:
HF_HUB_ENABLE_HF_TRANSFER: 1
SERVING_MIN_WORKERS: 1
OPTION_ENGINE: OnnxRuntime
HF_HOME: /tmp/.cache/huggingface
OMP_NUM_THREADS: 1
HF_TASK: text-embedding
SAGEMAKER_SAFE_PORT_RANGE: 14000-14999
SERVING_OPTS: -Dai.djl.logging.level=debug
SERVING_BATCH_SIZE: 32
HF_MODEL_ID: intfloat/multilingual-e5-base
SERVING_FEATURES: vllm,lmi-dist
SERVING_MAX_WORKERS: 1
DJL_CACHE_DIR: /tmp/.djl.ai
[INFO ] FolderScanPluginManager - scanning for plugins...
[INFO ] FolderScanPluginManager - scanning in plug-in folder :/opt/djl/plugins
[INFO ] FolderScanPluginManager - scanning in plug-in folder :/usr/local/djl-serving-0.31.0/plugins
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: cache-engines/jar:file:/usr/local/djl-serving-0.31.0/plugins/cache-0.31.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: kserve/jar:file:/usr/local/djl-serving-0.31.0/plugins/kserve-0.31.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: secure-mode/jar:file:/usr/local/djl-serving-0.31.0/plugins/secure-mode-0.31.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: console/jar:file:/usr/local/djl-serving-0.31.0/plugins/management-console-0.31.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: static-file-plugin/jar:file:/usr/local/djl-serving-0.31.0/plugins/static-file-plugin-0.31.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: plugin-management/jar:file:/usr/local/djl-serving-0.31.0/plugins/plugin-management-plugin-0.31.0.jar!/META-INF/plugin.definition
[INFO ] FolderScanPluginManager - Loading plugin: {console/jar:file:/usr/local/djl-serving-0.31.0/plugins/management-console-0.31.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin console changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {static-file-plugin/jar:file:/usr/local/djl-serving-0.31.0/plugins/static-file-plugin-0.31.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin static-file-plugin changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {plugin-management/jar:file:/usr/local/djl-serving-0.31.0/plugins/plugin-management-plugin-0.31.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin plugin-management changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {cache-engines/jar:file:/usr/local/djl-serving-0.31.0/plugins/cache-0.31.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin cache-engines changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {secure-mode/jar:file:/usr/local/djl-serving-0.31.0/plugins/secure-mode-0.31.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin secure-mode changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {kserve/jar:file:/usr/local/djl-serving-0.31.0/plugins/kserve-0.31.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin kserve changed state to INITIALIZED
[INFO ] PluginMetaData - plugin console changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin static-file-plugin changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin plugin-management changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin cache-engines changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin secure-mode changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin kserve changed state to ACTIVE reason: plugin ready
[INFO ] FolderScanPluginManager - 6 plug-ins found and loaded.
[DEBUG] ModelServer - Created serving.properties for model at path /tmp/.djl.ai/2236c1d9060d68fe528c80264239b9b676d60da3/serving.properties
[INFO ] ModelServer - Initializing model: intfloat_multilingual_e5_base=/tmp/.djl.ai/2236c1d9060d68fe528c80264239b9b676d60da3
[DEBUG] SimpleRepository - Skip prepare for local repository.
[DEBUG] LmiUtils - Found config file config.json in hub
[WARN ] LmiConfigRecommender - The model task architecture [XLMRobertaModel] is not supported for optimized inference. LMI will attempt to load the model using HuggingFace Accelerate. Optimized inference performance is only available for the following task architectures: [LMHeadModel, ForConditionalGeneration, ForCausalLM]
[INFO ] LmiUtils - Detected mpi_mode: null, rolling_batch: disable, tensor_parallel_degree: 8, for modelType: xlm-roberta
[INFO ] ModelInfo - M-0001: Apply per model settings:
job_queue_size: 1000
max_dynamic_batch_size: 32
max_batch_delay: 100
max_idle_time: 60
load_on_devices: *
engine: OnnxRuntime
mpi_mode: null
option.entryPoint: null
option.task: text-embedding
option.tensor_parallel_degree: 8
option.max_rolling_batch_size: 32
option.pipeline_parallel_degree: 1
option.model_id: intfloat/multilingual-e5-base
option.rolling_batch: disable
[INFO ] LmiUtils - Converting model to onnx artifacts
[INFO ] LmiUtils - Converting model to onnx artifacts: [djl-convert, --output-dir, /tmp/.djl.ai/onnx/2236c1d9060d68fe528c80264239b9b676d60da3, --output-format, OnnxRuntime, -m, intfloat/multilingual-e5-base, --optimize, O4, --device, cuda]
[DEBUG] LmiUtils - convert: converting HuggingFace hub model: intfloat/multilingual-e5-base
[DEBUG] LmiUtils - convert: Saving onnxruntime model: intfloat/multilingual-e5-base ...
[DEBUG] LmiUtils - convert: PyTorch version 2.5.1+cu124 available.
[DEBUG] LmiUtils - convert: Load pretrained SentenceTransformer: intfloat/multilingual-e5-base
[DEBUG] LmiUtils - convert: /usr/local/lib/python3.11/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py:115: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
[DEBUG] LmiUtils - convert: warnings.warn(
[DEBUG] LmiUtils - convert: Overridding for_gpu=False to for_gpu=True as half precision is available only on GPU.
[DEBUG] LmiUtils - convert: /usr/local/lib/python3.11/dist-packages/optimum/onnxruntime/configuration.py:779: FutureWarning: disable_embed_layer_norm will be deprecated soon, use disable_embed_layer_norm_fusion instead, disable_embed_layer_norm_fusion is set to True.
[DEBUG] LmiUtils - convert: warnings.warn(
[DEBUG] LmiUtils - convert: There is no gpu for onnxruntime to do optimization.
[DEBUG] LmiUtils - convert: Fused LayerNormalization: 25
[DEBUG] LmiUtils - convert: Fused Gelu: 12
[DEBUG] LmiUtils - convert: Removed 432 nodes
[DEBUG] LmiUtils - convert: Fused SkipLayerNormalization: 24
[DEBUG] LmiUtils - convert: symbolic shape inference disabled or failed.
[DEBUG] LmiUtils - convert: Fused Reshape: 2
[DEBUG] LmiUtils - convert: Removed 22 nodes
[DEBUG] LmiUtils - convert: Fused BiasGelu: 12
[DEBUG] LmiUtils - convert: Fused SkipLayerNormalization(add bias): 24
[DEBUG] LmiUtils - convert: Fused FastGelu(GeluApproximation): 12
[DEBUG] LmiUtils - convert: opset version: 14
[DEBUG] LmiUtils - convert: Failed to convert model: intfloat/multilingual-e5-base.
[DEBUG] LmiUtils - convert: [TypeInferenceError] Cannot infer type and shape for node name SkipLayerNorm_AddBias_0. No opset import for domain com.microsoft optype SkipLayerNormalization
[DEBUG] LmiUtils - convert: Traceback (most recent call last):
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/djl_converter/model_converter.py", line 74, in main
[DEBUG] LmiUtils - convert: result, reason, _ = converter.save_model(model_info, task, args,
[DEBUG] LmiUtils - convert: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/djl_converter/huggingface_converter.py", line 59, in save_model
[DEBUG] LmiUtils - convert: return self.save_onnx_model(model_info, task, args, temp_dir,
[DEBUG] LmiUtils - convert: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/djl_converter/huggingface_converter.py", line 91, in save_onnx_model
[DEBUG] LmiUtils - convert: main()
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/optimum/commands/optimum_cli.py", line 208, in main
[DEBUG] LmiUtils - convert: service.run()
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/optimum/commands/export/onnx.py", line 265, in run
[DEBUG] LmiUtils - convert: main_export(
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/optimum/exporters/onnx/__main__.py", line 373, in main_export
[DEBUG] LmiUtils - convert: onnx_export_from_model(
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/optimum/exporters/onnx/convert.py", line 1214, in onnx_export_from_model
[DEBUG] LmiUtils - convert: optimizer.optimize(save_dir=output, optimization_config=optimization_config, file_suffix="")
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/optimum/onnxruntime/optimization.py", line 211, in optimize
[DEBUG] LmiUtils - convert: optimizer.convert_float_to_float16(
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/onnxruntime/transformers/onnx_model.py", line 796, in convert_float_to_float16
[DEBUG] LmiUtils - convert: fp16_model = convert_float_to_float16(model, **parameters)
[DEBUG] LmiUtils - convert: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/onnxruntime/transformers/float16.py", line 257, in convert_float_to_float16
[DEBUG] LmiUtils - convert: model = func_infer_shape(model)
[DEBUG] LmiUtils - convert: ^^^^^^^^^^^^^^^^^^^^^^^
[DEBUG] LmiUtils - convert: File "/usr/local/lib/python3.11/dist-packages/onnx/shape_inference.py", line 46, in infer_shapes
[DEBUG] LmiUtils - convert: inferred_model_str = C.infer_shapes(
[DEBUG] LmiUtils - convert: ^^^^^^^^^^^^^^^
[DEBUG] LmiUtils - convert: onnx.onnx_cpp2py_export.shape_inference.InferenceError: [TypeInferenceError] Cannot infer type and shape for node name SkipLayerNorm_AddBias_0. No opset import for domain com.microsoft optype SkipLayerNormalization
[ERROR] ModelServer - Failed register workflow
java.util.concurrent.CompletionException: ai.djl.engine.EngineException: Model conversion process failed!
#011at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
#011at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) [?:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) [?:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760) [?:?]
#011at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
#011at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) [?:?]
#011at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
#011at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
#011at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) [?:?]
Caused by: ai.djl.engine.EngineException: Model conversion process failed!
#011at ai.djl.serving.wlm.LmiUtils.convertOnnx(LmiUtils.java:216) ~[wlm-0.31.0.jar:?]
#011at ai.djl.serving.wlm.LmiUtils.convertOnnxModel(LmiUtils.java:163) ~[wlm-0.31.0.jar:?]
#011at ai.djl.serving.wlm.ModelInfo.initialize(ModelInfo.java:562) ~[wlm-0.31.0.jar:?]
#011at ai.djl.serving.models.ModelManager.lambda$registerWorkflow$2(ModelManager.java:105) ~[serving-0.31.0.jar:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
#011... 6 more
[INFO ] ModelServer - Model server stopped.
[ERROR] ModelServer - Unexpected error
ai.djl.serving.http.ServerStartupException: Failed to initialize startup models and workflows
#011at ai.djl.serving.ModelServer.start(ModelServer.java:205) ~[serving-0.31.0.jar:?]
#011at ai.djl.serving.ModelServer.startAndWait(ModelServer.java:160) ~[serving-0.31.0.jar:?]
#011at ai.djl.serving.ModelServer.main(ModelServer.java:129) [serving-0.31.0.jar:?]
Caused by: java.util.concurrent.CompletionException: ai.djl.engine.EngineException: Model conversion process failed!
#011at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
#011at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) ~[?:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
Caused by: ai.djl.engine.EngineException: Model conversion process failed!
#011at ai.djl.serving.wlm.LmiUtils.convertOnnx(LmiUtils.java:216) ~[wlm-0.31.0.jar:?]
#011at ai.djl.serving.wlm.LmiUtils.convertOnnxModel(LmiUtils.java:163) ~[wlm-0.31.0.jar:?]
#011at ai.djl.serving.wlm.ModelInfo.initialize(ModelInfo.java:562) ~[wlm-0.31.0.jar:?]
#011at ai.djl.serving.models.ModelManager.lambda$registerWorkflow$2(ModelManager.java:105) ~[serving-0.31.0.jar:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
#011at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
#011at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
```

## How to Reproduce?

### Steps to reproduce
(Paste the commands you ran that produced the error.)

1. Run the above python script

## What have you tried to solve it?

1. Using larger instance type

## Environment Info

Please run the command `./gradlew debugEnv` from the root directory of DJL (if necessary, clone DJL first). It will output information about your system, environment, and installation that can help us debug your issue. Paste the output of the command below:

N/A