NVIDIA / NVIDIA/NeMo-Retriever
[BUG]: nv-ingest-nv-ingest-ms-runtime-1 container failed to connect yolox endpoints
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Version
24.08
Which installation method(s) does this occur on?
Docker
Describe the bug.
nv-ingest-nv-ingest-ms-runtime-1 container failed to run with error as failed to connect yolox endpoints
Docker Containers Status
ubuntu@vm:~/nv-ingest$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
df6d0820de49 nvcr.io/ohlfw0olaadg/ea-participants/nv-ingest:24.08 "/opt/conda/bin/tini…" 11 seconds ago Exited (1) 1 second ago nv-ingest-nv-ingest-ms-runtime-1
bbba107092e1 otel/opentelemetry-collector-contrib:0.91.0 "/otelcol-contrib --…" About a minute ago Up 44 seconds 0.0.0.0:4317-4318->4317-4318/tcp, :::4317-4318->4317-4318/tcp, 0.0.0.0:8888-8889->8888-8889/tcp, :::8888-8889->8888-8889/tcp, 0.0.0.0:13133->13133/tcp, :::13133->13133/tcp, 55678/tcp, 0.0.0.0:32769->9411/tcp, [::]:32769->9411/tcp, 0.0.0.0:55680->55679/tcp, [::]:55680->55679/tcp nv-ingest-otel-collector-1
f75619d3f126 nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1 "/opt/nvidia/nvidia_…" About a minute ago Up 45 seconds 0.0.0.0:8012->8000/tcp, [::]:8012->8000/tcp, 0.0.0.0:8013->8001/tcp, [::]:8013->8001/tcp, 0.0.0.0:8014->8002/tcp, [::]:8014->8002/tcp nv-ingest-embedding-1
9418a2ab08e7 nvcr.io/ohlfw0olaadg/ea-participants/cached:0.1.0 "/opt/nvidia/nvidia_…" About a minute ago Up 45 seconds 0.0.0.0:8006->8000/tcp, [::]:8006->8000/tcp, 0.0.0.0:8007->8001/tcp, [::]:8007->8001/tcp, 0.0.0.0:8008->8002/tcp, [::]:8008->8002/tcp nv-ingest-cached-1
40a89864219e nvcr.io/ohlfw0olaadg/ea-participants/paddleocr:0.1.0 "/opt/nvidia/nvidia_…" About a minute ago Up 45 seconds 0.0.0.0:8009->8000/tcp, [::]:8009->8000/tcp, 0.0.0.0:8010->8001/tcp, [::]:8010->8001/tcp, 0.0.0.0:8011->8002/tcp, [::]:8011->8002/tcp nv-ingest-paddle-1
3d98a8921104 nvcr.io/ohlfw0olaadg/ea-participants/nv-yolox-structured-images-v1:0.1.0 "/opt/nvidia/nvidia_…" About a minute ago Up 45 seconds 0.0.0.0:8000-8002->8000-8002/tcp, :::8000-8002->8000-8002/tcp nv-ingest-yolox-1
31012ffe11e4 nvcr.io/ohlfw0olaadg/ea-participants/deplot:1.0.0 "/opt/nvidia/nvidia_…" About a minute ago Up 45 seconds 0.0.0.0:8003->8000/tcp, [::]:8003->8000/tcp, 0.0.0.0:8004->8001/tcp, [::]:8004->8001/tcp, 0.0.0.0:8005->8002/tcp, [::]:8005->8002/tcp nv-ingest-deplot-1
d27e876c2ec1 openzipkin/zipkin "start-zipkin" About a minute ago Up 46 seconds (healthy) 9410/tcp, 0.0.0.0:9411->9411/tcp, :::9411->9411/tcp nv-ingest-zipkin-1
b0e7456e16ff redis/redis-stack "/entrypoint.sh" About a minute ago Up 45 seconds 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp, 8001/tcp nv-ingest-redis-1
6764b4f11b6a grafana/grafana "/run.sh" About a minute ago Up 46 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana-service
bd969769645a prom/prometheus:latest "/bin/prometheus --w…" About a minute ago Up 45 seconds 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp nv-ingest-prometheus-1
Minimum reproducible example
Followed the QuickStart instructions
Relevant log output
Docker Container Logs
ubuntu@vm:~/nv-ingest$ docker logs nv-ingest-nv-ingest-ms-runtime-1
/opt/conda/envs/morpheus/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Configuration loaded and validated: image_caption_extraction_module={} image_storage_module={} metadata_injection_module={} otel_meter_module={} pdf_extractor_module={} redis_task_sink={} redis_task_source={} text_splitting_module={}
INFO:__main__:Starting pipeline setup
INFO:__main__:MESSAGE_CLIENT_HOST: redis
INFO:__main__:MESSAGE_CLIENT_PORT: 6379
INFO:__main__:TABLE_DETECTION_GRPC_TRITON:
INFO:__main__:TABLE_DETECTION_HTTP_TRITON:
INFO:__main__:PADDLE_GRPC_ENDPOINT: paddle:8001
INFO:__main__:PADDLE_HTTP_ENDPOINT: http://paddle:8000/v1/infer
INFO:__main__:DEPLOT_GRPC_ENDPOINT: ""
INFO:__main__:DEPLOT_HTTP_ENDPOINT: http://deplot:8000/v1/chat/completions
INFO:__main__:CACHED_GRPC_ENDPOINT: cached:8001
INFO:__main__:CACHED_HTTP_ENDPOINT: http://cached:8000/v1/infer
Traceback (most recent call last):
File "/workspace/pipeline.py", line 710, in <module>
cli()
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/workspace/pipeline.py", line 706, in cli
pipeline(morpheus_pipeline_config, final_ingest_config)
File "/workspace/pipeline.py", line 584, in pipeline
setup_ingestion_pipeline(pipe, morpheus_pipeline_config, ingest_config)
File "/workspace/pipeline.py", line 532, in setup_ingestion_pipeline
pdf_extractor_stage = add_pdf_extractor_stage(pipe, morpheus_pipeline_config, ingest_config, default_cpu_count)
File "/workspace/pipeline.py", line 251, in add_pdf_extractor_stage
generate_pdf_extractor_stage(
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/nv_ingest/stages/pdf_extractor_stage.py", line 135, in generate_pdf_extractor_stage
validated_config = PDFExtractorSchema(**extractor_config)
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for PDFExtractorSchema
pdfium_config -> __root__
Both gRPC and HTTP services cannot be empty for yolox_endpoints. (type=value_error)
ubuntu@vm:~/nv-ingest$ docker logs nv-ingest-yolox-1 | tail -10f
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.483885 123 infer_handler.h:1322] "Thread started for ModelInferHandler""}
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.484005 123 infer_handler.cc:680] "New request handler for ModelInferHandler, 0""}
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.484029 123 infer_handler.h:1322] "Thread started for ModelInferHandler""}
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.484210 123 stream_infer_handler.cc:128] "New request handler for ModelStreamInferHandler, 0""}
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.484275 123 infer_handler.h:1322] "Thread started for ModelStreamInferHandler""}
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.484283 123 grpc_server.cc:2463] "Started GRPCInferenceService at 0.0.0.0:8001""}
{"timestamp": "2024-10-15 19:04:20,486", "level": "ERROR", "message": "I1015 19:04:20.484439 123 http_server.cc:4692] "Started HTTPService at 0.0.0.0:8080""}
{"timestamp": "2024-10-15 19:04:20,528", "level": "ERROR", "message": "I1015 19:04:20.525731 123 http_server.cc:362] "Started Metrics Service at 0.0.0.0:8002""}
{"timestamp": "2024-10-15 19:04:21,659", "level": "INFO", "message": "Starting NIM inference server"}
{"timestamp": "2024-10-15 19:04:21,674", "level": "INFO", "message": "Serving HTTP server on port 8000"}
Other/Misc.
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the QuickStart and /workspace/pipeline.py, especially add_pdf_extractor_stage, then inspect the PDF extractor configuration and the YOLOX endpoint settings shown in the logs. Reproduce with Docker and verify that the runtime container starts without the PDFExtractorSchema validation error and can connect to the YOLOX service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100