NVIDIA / NVIDIA/nvcf

fix(function-autoscaler): fall back to control-plane metrics after worker metric failure

Open
#1,379 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

The Function Autoscaler selects a metrics source for each function version and
caches that choice. It currently falls back from worker-thread metrics only
when the initial worker-count query returns no series. After the worker source
is cached, a later empty result keeps the worker source with an instance count
of zero. A later query error also keeps the worker source with an instance
count of zero.

Update the runtime fallback behavior so a function version can switch to
control-plane metrics when worker metrics become unavailable after initial
source selection. This prevents a later worker-metrics outage from being
treated as zero current instances.

Define how and when the autoscaler retries worker metrics after fallback. The
transition should avoid rapid source flapping.

Definition of Done

  • A cached worker-thread source falls back to control-plane metrics when the
    worker-count query returns no series.
  • A cached worker-thread source falls back to control-plane metrics when the
    worker-count query fails.
  • The failed worker query is not treated as zero current instances when
    control-plane metrics are available.
  • Source-cache behavior supports a documented recovery path without rapid
    switching between sources.
  • Tests cover a later empty worker result, a later worker query error, the
    control-plane fallback, and the existing successful worker path.
  • Logs identify the source transition with the function ID, function version
    ID, and NCA ID.
  • Autoscaling documentation describes fallback after initial source selection
    and the recovery behavior.

Resources (optional)


By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/control-plane-services/function-autoscaler/crates/server/src/work/mod.rs and review the linked finding and related documentation pull request. Trace cached worker-source selection and worker-count errors, then define a non-flapping recovery path; done means fallback, transition logs, tests for empty and failed later queries, the successful worker path, and updated autoscaling documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.