NVIDIA / NVIDIA/NeMo-Retriever

Fix service-mode Compose config drift and NIM startup portability

Open
#2,424 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Summary

nemo_retriever/dev/compose/service-mode.compose.yaml still generates the
deprecated local_models.extract.use_graphic_elements setting. The current
ServiceConfig forbids that extra key, so the retriever container enters a
restart loop before the service can start.

The same self-hosted NIM validation exposed three additional portability
problems in the current Compose stack: an unconditional named nvidia runtime
assumption and model-store volume ownership that is incompatible with the
non-root NIM containers, plus repeated OTLP export failures from the embedding
NIM despite telemetry being disabled.

These were discovered while validating #2421 against current upstream/main.
They are Compose/service startup issues, not part of the local OCR batching
change.

Observed failures

1. Deprecated graphic-elements setting prevents service startup

The generated configuration contains:

local_models:
  extract:
    use_graphic_elements: ${LOCAL_EXTRACT_USE_GRAPHIC_ELEMENTS:-true}

Current LocalExtractConfig does not define this deprecated field and uses
extra="forbid". The retriever repeatedly exits with:

local_models.extract.use_graphic_elements
  Extra inputs are not permitted [type=extra_forbidden, input_value=True]

This occurs even with LOCAL_MODELS_ENABLED=false, because the full generated
configuration is validated at startup.

2. Compose assumes a named NVIDIA runtime

The base Compose file sets runtime: nvidia. On the validation host,
docker run --gpus ... and Compose GPU device reservations work, but the
daemon does not register a named nvidia runtime. A temporary override to
runtime: runc retained working GPU reservations and allowed all four core
NIMs to become healthy.

The stack should use one portable GPU-selection mechanism or document and
validate the named-runtime prerequisite.

3. Named model-store volumes are not writable by the NIM users

The page-elements, table-structure, and OCR NIM containers run as non-root
users. Fresh root-owned named volumes mounted at /model-store prevented those
containers from creating their model-store directories. Replacing them with
writable bind mounts allowed startup.

Volume initialization should establish ownership/permissions compatible with
the container user without requiring host-side chmod or ad hoc bind mounts.

4. Disabled telemetry still targets an absent collector

The embedding NIM remained healthy and served requests, but repeatedly logged:

[OTLP HTTP Client] Session state: connection failed.
Could not resolve host: otel-collector

This occurred with NIM_ENABLE_OTEL=false because the Compose environment
still provides TRITON_OTEL_URL=http://otel-collector:4318/v1/traces. The core
NIM-only profile does not start that collector. Disabling telemetry should
also omit the exporter target, or the collector should be included whenever
the target is configured.

Reproduction context

  • Source: upstream/main commit 3d9e26f1a2d2fd73af499bb7a9ef7fe855739841
  • Command surface: docker compose --profile nims-core -f nemo_retriever/dev/compose/service-mode.compose.yaml up -d
  • Page/table image: nvcr.io/nim/nvidia/nemotron-object-detection:2.0.0
  • OCR image: nvcr.io/nim/nvidia/nemotron-ocr-v2:2.0.0
  • Embedding image: nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0
  • Related experiment record: #2421

With temporary overrides for the three startup issues, the four NIMs, vector database,
and standalone retriever became healthy, and a 1,360-page ViDoRe service-mode
run completed successfully. The telemetry failure was non-fatal but continued
through a subsequent 19,252-page one-GPU NIM run.

Acceptance criteria

  • Remove the deprecated use_graphic_elements environment/config output from the Compose-generated service configuration.
  • Add a lightweight validation that the generated Compose service configuration loads through the current ServiceConfig.
  • Make GPU runtime selection portable across daemons that support device reservations but do not register a named nvidia runtime, or fail early with a precise documented prerequisite.
  • Ensure fresh page-elements, table-structure, and OCR model-store volumes are writable by their container users.
  • When telemetry is disabled, do not configure an unreachable OTLP exporter target or emit repeated connection errors.
  • Verify the core NIM profile reaches healthy state without a temporary Compose override.
  • Verify retriever and vector database reach healthy state in standalone service mode.

Scope

This issue is limited to the self-hosted Compose startup path. It should not
change OCR batching policy, NIM batch-size defaults, #2323 behavior, or
benchmark semantics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nemo_retriever/dev/compose/service-mode.compose.yaml and run the documented nims-core Compose command, then inspect how its generated configuration is validated by ServiceConfig. Check the Compose GPU runtime, model-store volumes, and telemetry environment alongside the acceptance criteria. Done means the generated config validates and the core NIM, retriever, and vector database services become healthy without temporary overrides.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.