NVIDIA / NVIDIA/nvcf

Wire Dynamo workers to ModelExpress in the sample chart

Open
#1,346 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

Add the NVCF function chart: a DynamoGraphDeployment whose vLLM workers load
weights through ModelExpress, with a switch to run the identical topology on the
native loader for comparison.

Provider-neutral by default. Fabric-specific settings live in a separate override
file, so the chart carries no cloud-specific values.

Definition of Done

  • Worker replica count is the scale-out dimension, and the chart renders
    correctly at 2, 4, and 10 replicas.
  • vLLM load format names the ModelExpress loader. Environment variables alone
    configure a loader vLLM never calls, which looks like success while
    ModelExpress does nothing.
  • The client server address is set under both the current and the older variable
    name, since the 0.4.0 client reads both.
  • The vLLM plugin is named explicitly, because the delivered vLLM is below 0.23.
  • The NIXL backend is configurable, and the fabric-specific choice is documented
    with the reason.
  • Disabling ModelExpress emits no ModelExpress environment at all, so the
    baseline is a clean comparison.
  • No plaintext secret renders. The model token is optional, since workers reading
    from a peer do not need one.
  • helm lint passes and helm template is deterministic.
  • An override sample carries the EFA settings and the untested InfiniBand and
    RoCE settings, clearly labelled.

Resources

Backend choice, which is the part most likely to be got wrong. The client accepts
UCX or LIBFABRIC and defaults to UCX. AWS EFA needs LIBFABRIC together
with the vllm-runtime:1.2.1-efa-amd64 image, because the stock image ships no
libfabric.so.1 and the plugin cannot load without it. This is the combination
that was validated end to end.

The UCX default carries a known defect. On this client version the source
worker segfaulted while serving a transfer, in nixlUcxSharedThread::run(), and
it reproduced over plain TCP, so it is not fabric-specific. That is disclosed in
the chart values, the override sample, and the guide, because InfiniBand and RoCE
users take the same path and neither fabric was available to test.

Three deployment failures worth encoding, all hit in practice:

  • vpc.amazonaws.com/efa must be declared under limits, not requests.
    Kubernetes rejects a Pod that requests an extended resource without a matching
    limit, and the Dynamo operator passes the block through verbatim. The result is
    ERR_CREATE_POD and pods that never appear, which reads as a scheduling
    problem rather than a spec problem.
  • EFA caps workers at one per node. G-family instances expose a single EFA
    interface and the device plugin advertises one unit per node, so replicas must
    scale with nodes rather than with GPUs per node.
  • The locked-memory limit cannot be fixed from the chart. It is inherited from
    the node's container runtime, and CAP_IPC_LOCK does not help because the
    image runs as UID 1000 and a non-root process has an empty effective capability
    set.

Existing structure reference: examples/function-samples/helmchart-samples/dynamo-operator-sample/
Override convention: examples/function-samples/helmchart-samples/multi-node-helm-function-test/override.yaml.sample

Part of #1113.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing examples/function-samples/helmchart-samples/dynamo-operator-sample/ with examples/function-samples/helmchart-samples/multi-node-helm-function-test/override.yaml.sample. Trace the chart values and templates for replica scaling, loader configuration, secrets, NIXL, and fabric overrides, then run helm lint and deterministic helm template renders. Done means the required topologies, clean disabled baseline, safe secret handling, documented overrides, and validation checks all pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.