Standardize optional HPA support for eligible NVCF control-plane services

Is this related to a problem? Please describe.

NVCF operators need a supported way to scale control-plane services as API, invocation, and operational load grows.

Horizontal Pod Autoscaler (HPA) support is currently inconsistent across control-plane charts:

• Some services include conditional autoscaling/v2 HPA resources.
• Some charts expose autoscaling values without rendering an HPA.
• Other services have no documented autoscaling contract.
• Certain singleton, stateful, or topology-sensitive components cannot safely scale horizontally.

Operators can manage replicas or external HPA resources themselves, but this requires service-specific knowledge and can conflict with Helm-managed replica counts.

This is separate from [function autoscaling tracked in #15](https://github.com/NVIDIA/nvcf/issues/15). This request concerns Kubernetes replicas for services running in the control-plane cluster.

Describe the solution you'd like

Define a consistent, optional HPA contract for NVCF control-plane services that are verified to be horizontally safe.

The implementation should:

1. Classify each control-plane service as:

   • HPA supported.
   • HPA conditionally supported, with documented constraints.
   • HPA unsupported, with the reason and recommended scaling procedure.

2. Provide consistent autoscaling configuration for supported charts:

service:
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 80
    # targetMemoryUtilizationPercentage: 80
    # behavior: {}

3. When autoscaling is enabled:

   • Render an autoscaling/v2 HorizontalPodAutoscaler.
   • Target the correct Deployment.
   • Omit the Deployment’s spec.replicas field so Helm does not reset HPA decisions.
   • Validate required resource requests and replica bounds.

4. Preserve existing fixed-replica behavior when autoscaling is disabled.

5. Keep autoscaling disabled by default until each service has been validated.

6. Document metrics prerequisites, supported topologies, safe defaults, troubleshooting, and the distinction between HPA and NVCF function autoscaling.

7. Add Helm rendering tests and an end-to-end scaling test for at least one supported control-plane service.

Describe alternatives you've considered

Manual replica management

Operators can monitor services and update replica counts manually. This remains a useful fallback but requires human intervention and does not respond consistently to changing load.

Operator-maintained HPA resources

Operators can create HPAs outside the NVCF charts. This is flexible but duplicates service-specific configuration, can conflict with Helm, and makes unsupported configurations easier to deploy.

Custom metrics, KEDA, or VictoriaMetrics-backed scaling

Application-level signals may eventually provide better scaling decisions than CPU or memory. They also require additional metric adapters and design work. The initial implementation could use Kubernetes resource metrics while leaving room for custom metrics later.

Vertical Pod Autoscaler or static sizing guidance

VPA and sizing guidance can improve resource allocation, but they do not replace horizontal scaling for services that safely support multiple replicas.

Additional context

Existing repository references:

• [Control-plane operations](https://github.com/NVIDIA/nvcf/blob/main/docs/user/control-plane-operations.md)
• [Control-plane observability](https://github.com/NVIDIA/nvcf/blob/main/docs/user/observability.md)
• [gRPC proxy HPA template](https://github.com/NVIDIA/nvcf/blob/main/deploy/helm/grpc-proxy/grpc-proxy/templates/hpa.yaml)
• [Rate limiter HPA template](https://github.com/NVIDIA/nvcf/blob/main/deploy/helm/ratelimiter/nvcf-ratelimiter/templates/hpa.yaml)
• [Helm reval autoscaling values](https://github.com/NVIDIA/nvcf/blob/main/deploy/helm/helm-reval/values.yaml)
• [gRPC topology restrictions](https://github.com/NVIDIA/nvcf/blob/main/docs/user/grpc-invocation-enablement.md)
• [Function Autoscaler versus HPA](https://github.com/NVIDIA/nvcf/blob/main/docs/user/autoscaling/index.md#function-autoscaler-vs-horizontal-pod-autoscaler)

The initial implementation should not:

• Apply HPA automatically to every control-plane component.
• Autoscale stateful or singleton services without a component-specific design.
• Remove documented single-replica topology restrictions.
• Require custom metrics infrastructure.
• Perform control-plane node autoscaling or infrastructure provisioning.

---

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.

Start by comparing deploy/helm/grpc-proxy/grpc-proxy/templates/hpa.yaml, deploy/helm/ratelimiter/nvcf-ratelimiter/templates/hpa.yaml, and deploy/helm/helm-reval/values.yaml, then read the control-plane operations and observability docs. Done means eligible services have a documented HPA classification and consistent optional Helm behavior, with rendering coverage and an end-to-end scaling test for at least one supported service.