NVIDIA / NVIDIA/nvcf

Expose the LLM request router to split-cluster workers

Open
#689 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Describe the bug

The self-managed multi-cluster topology currently advertises llm-request-router.nvcf.svc.cluster.local:50071 to workers. That address is local to the control-plane cluster and is not resolvable or reachable from a separate compute cluster.

The local multi-cluster tooling has no LLM request-router alias Service or Endpoints in the compute cluster, no port-50071 k3d mapping, and no Gateway listener or TCPRoute for the router. The current BDD does not launch an LLM worker, so it does not exercise this connection.

Expected behavior

Split-cluster workers receive an LLM request-router address that resolves and is reachable from the compute cluster, and local multi-cluster tests verify the connection path.

Suggested scope

  • Expose TCP port 50071 from the control-plane cluster through a Gateway and TCPRoute, load balancer, or equivalent supported path.
  • Add the required k3d host-port plumbing for local multi-cluster testing.
  • Add a compute-cluster alias Service and Endpoints, or define and document an external DNS endpoint.
  • Keep tests/bdd/fixtures/self-managed-local-bdd-multi.yaml and the raw seed in tests/bdd/godog_test.go aligned.
  • Add an end-to-end test that launches an LLM worker across clusters.
  • Document the split-cluster network, DNS, routing, and TLS requirements.

Context

Follow-up to #678. That pull request wires the configured address through the NVCF API, but intentionally does not add cross-cluster network exposure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/bdd/fixtures/self-managed-local-bdd-multi.yaml and the raw seed in tests/bdd/godog_test.go, then trace the existing split-cluster routing setup described in the issue. Verify the selected exposure path, k3d port mapping, compute-cluster aliasing, and required DNS, routing, and TLS behavior. Done means an LLM worker connects across clusters and the end-to-end BDD test passes with the network requirements documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.