Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer

I am experiencing an issue where the nemo-retriever-embedding-microservice container fails to download the required model files during startup. Here are the details:

Command:
$ docker compose -f docker-compose.yaml -f ../../local_deploy/docker-compose-nim-ms.yaml -f ../../local_deploy/docker-compose-vectordb.yaml --profile local-nim up

Error Logs:

nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | =========================================
nemo-retriever-embedding-microservice | == NVIDIA Retriever Text Embedding NIM ==
nemo-retriever-embedding-microservice | =========================================
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | NVIDIA Release 1.0.1 (build 0cf1c7ad8e51bdffc4e4d4226735dbb7f59d70d4)
nemo-retriever-embedding-microservice | Model: nvidia/nv-embedqa-e5-v5
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | Container image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
nemo-retriever-embedding-microservice | This NIM container is governed by the NVIDIA AI Product Agreement here:
nemo-retriever-embedding-microservice | https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.
nemo-retriever-embedding-microservice | A copy of this license can be found under /opt/nim/LICENSE.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | The use of this model is governed by the AI Foundation Models Community License
nemo-retriever-embedding-microservice | here: https://docs.nvidia.com/ai-foundation-models-community-license.pdf.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | OTEL Logging handler requested, but Python logging auto-instrumentation not set up. Set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to enable logging auto-instrumentation.
nemo-retriever-embedding-microservice | downloading nim/nvidia/nv-embedqa-e5-v5:5_tokenizer
nemo-retriever-embedding-microservice | This could take a while.
nemo-retriever-embedding-microservice | --2025-01-22 23:34:03-- https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nv-embedqa-e5-v5/versions/5_tokenizer/zip
nemo-retriever-embedding-microservice | Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 54.68.27.61, 44.227.231.63
nemo-retriever-embedding-microservice | Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|54.68.27.61|:443... connected.
nemo-retriever-embedding-microservice | HTTP request sent, awaiting response... 410 Gone
nemo-retriever-embedding-microservice | 2025-01-22 23:34:04 ERROR 410: Gone.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | 2025-01-22T23:34:04Z WARNING: tools.nim.ngc_models - Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer: Failure from NGC CLI -
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice exited with code 1

Environment:

• Docker Compose Version: v2.5.0

Issue:
The nemo-retriever-embedding-microservice service uses the image nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1. However, during startup, it attempts to download the resource nim/nvidia/nv-embedqa-e5-v5:5_tokenizer from NVIDIA NGC. The request fails with a 410 Gone error, indicating that the resource might no longer be available.

Questions:
1.Why is the nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1 image trying to download the 5_tokenizer resource?
2.Has this resource (nim/nvidia/nv-embedqa-e5-v5:5_tokenizer) been deprecated or removed from NGC?
3.Should I update the image version or modify the service configuration to resolve this issue?
4.Is there an alternative approach to pre-download the required models and avoid runtime dependency issues?
Any guidance on resolving this issue would be greatly appreciated.

Thank you!

Start with docker-compose.yaml, ../../local_deploy/docker-compose-nim-ms.yaml, and the nv-embedqa-e5-v5:1.0.1 image configuration. Check why startup requests nim/nvidia/nv-embedqa-e5-v5:5_tokenizer and whether the referenced NGC resource and image version remain compatible. Done means the service starts successfully with supported model resources or the required configuration and supported version are documented.