Question about scaling with AlertManager

My actions before raising this issue

• Followed the troubleshooting guide
• Read/searched the docs
• Searched past issues

Openfaas uses prometheus to monitor function calls, and when function QPS is higher than some threshold autoscale will be triggered.

But after functions are scaled up, the QPS won't go down, so functions will still be scaled up until maxReplicas are reached.

In my opinion, when we scale up functions, the QPS for each function replica will go down, it means the load for each replica will go down.

So when we scale function to X replicas where QPS/X is relatively small, we can stop scale up.

Also when the alert is stop, replicas will be set to minReplicas, QPS per replica will arise and probabily higher than we'd expect

Expected Behaviour

1. When APIHighInvocationRate alert is fired, function should only scale up to some scale not maxReplicas.

2. when APIHighInvocationRate is stopped, we should scale down function gracefully just like we scale up, little by little, to finnaly reach a safe QPS per replica

Current Behaviour

1. When APIHighInvocationRate alert keeps firing (function QPS is high), function replicas will soon reach maxReplicas (default 20)

2. When APIHighInvocationRate alert stops, function replica will drop to minReplicas (default 1)

Possible Solution

1. To solve scale up issue, we could change prometheus alert rule, use QPS/replicas. In my local test I use:

sum by(function_name) (rate(gateway_function_invocation_total{code="200"}[10s]) / ignoring(code) gateway_service_count) > 5

2. To solve scale down issue, we could add a new scale-down endpoint in gateway and add a new prometheus rule to invoke scale-down api when replicas are more than we want

Steps to Reproduce (for bugs)

1. start minikube, deploy faas-netes and deploy some functions for future test.
2. invoke function 5+ times per second, I use hey to invoke curl function 6 times per second.

hey -m POST -q 6 -c 1 -d http://some-test-service:8080/ -z 30m http://192.168.99.100:31112/function/curl

3. kubectl logs -f deploy/gateway -c gateway -n openfaas| grep Scale to watch scale up/down logs

Start with the APIHighInvocationRate alert, the proposed Prometheus rule, and the gateway scaling logs described in the issue; reproduce the behavior using minikube, faas-netes, kubectl, and the provided hey command. Done means the scaling behavior matches both expected cases: bounded scale-up and gradual scale-down to a safe QPS per replica.