NVIDIA / NVIDIA/nvcf

Add preference-aware cluster placement for multi-cluster autoscaling

Open
#1,068 0 comments 0 reactions 2 assignees View on GitHub

@borao is already working on this.

Since Aug 25, 2026.

roadmap
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

NVCF currently sends new instance requests to eligible compute clusters, and
any cluster with capacity creates the instance. This can result in suboptimal
placement based on latency, power availability, cost, or cache locality.

Add preference-aware placement so compute clusters are considered in a defined
order. The NVCF Function Autoscaler will help determine this order using
per-cluster or per-function inputs.

An initial approach could attempt eligible compute clusters sequentially, in
descending order of preference.

Placement should align with LLM Gateway routing preferences so instances are
created where requests are expected to be routed.

Definition of Done

  • Aggregated historical data is analyzed to estimate the potential benefit.
  • Preference inputs, ownership, request flow, timeouts, and fallback behavior are defined.
  • The NVCF Function Autoscaler can contribute per-cluster or per-function preferences.
  • Eligible compute clusters are considered in a deterministic order.
  • Existing capacity, health, quota, and placement constraints remain authoritative.
  • Requests fall back when a preferred compute cluster cannot accept them.
  • Placement aligns with applicable LLM Gateway routing preferences.
  • Logs and metrics identify the selected cluster and fallback reason.
  • Tests cover preferred placement, fallback, missing preferences, and unavailable capacity.

Proposed Tasks

  • Analyze historical placement opportunities.
  • Define NVCF Function Autoscaler preference inputs.
  • Implement preference-aware compute-cluster selection.
  • Align placement with LLM Gateway routing.
  • Add observability, tests, and documentation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.