fix(function-autoscaler): fall back to control-plane metrics after worker metric failure
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Description
The Function Autoscaler selects a metrics source for each function version and
caches that choice. It currently falls back from worker-thread metrics only
when the initial worker-count query returns no series. After the worker source
is cached, a later empty result keeps the worker source with an instance count
of zero. A later query error also keeps the worker source with an instance
count of zero.
Update the runtime fallback behavior so a function version can switch to
control-plane metrics when worker metrics become unavailable after initial
source selection. This prevents a later worker-metrics outage from being
treated as zero current instances.
Define how and when the autoscaler retries worker metrics after fallback. The
transition should avoid rapid source flapping.
Definition of Done
- A cached worker-thread source falls back to control-plane metrics when the
worker-count query returns no series. - A cached worker-thread source falls back to control-plane metrics when the
worker-count query fails. - The failed worker query is not treated as zero current instances when
control-plane metrics are available. - Source-cache behavior supports a documented recovery path without rapid
switching between sources. - Tests cover a later empty worker result, a later worker query error, the
control-plane fallback, and the existing successful worker path. - Logs identify the source transition with the function ID, function version
ID, and NCA ID. - Autoscaling documentation describes fallback after initial source selection
and the recovery behavior.
Resources (optional)
- Review finding: https://github.com/NVIDIA/nvcf/pull/1020#discussion_r3846968605
- Related documentation Pull Request: https://github.com/NVIDIA/nvcf/pull/1020
- Current implementation:
src/control-plane-services/function-autoscaler/crates/server/src/work/mod.rs
By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/control-plane-services/function-autoscaler/crates/server/src/work/mod.rs and review the linked finding and related documentation pull request. Trace cached worker-source selection and worker-count errors, then define a non-flapping recovery path; done means fallback, transition logs, tests for empty and failed later queries, the successful worker path, and updated autoscaling documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100