Activator will OOM under backpressure
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
## What version of Knative?
0.22.3
## Expected Behavior
Activator should be resilient under backpressure
## Actual Behavior
Activator OOM'ed under backpressure
## Steps to Reproduce the Problem
1. Create an example target ksvc to mimic service with limited capacity (due to resource limitation, runtime/dependency bottleneck, or others). For the example here, I created a service with `maxReplicas=1, containerConcurrency=2`, and for each request it will sleep for 30s before returning a dummy response.
2. Attack the target ksvc with a throughput higher than it could handle. Example to attack with 200 rps with 1s timeout
```
ali --body 'dummy request body' -m POST --duration 7200s --rate 200 --timeout 1s https://ksvc-endpoint-url
```
3. Monitor activator resource usage: memory will heat up over time and eventually OOM. Increasing memory will only delay the OOM but not prevent it.
The default 600MB activator deployment OOM'ed after ~15 min.

Increased activator deployment memory to 2000MB. OOM'ed after ~50 min.

CPU utilization is low all through the period so activator is not scaled up (also not sure if scaling up activator can help here)
Is there any way/setting we can leverage to prevent activator from OOM under such backpressure, which could impact all ksvc in the cluster. Ideally if activator can deny surplus requests to the target ksvc if its queue is already long/full. Or if there's any tool we can borrow from Istio to enforce e.g. rate limiting on a per ksvc level? It looks like we do have a [breakerQueueDepth](https://github.com/knative/serving/blob/35abde024a891ba10565be82dfa167196324056d/pkg/activator/net/throttler.go#L56) but somehow it's not effective in terms of preventing OOM. Suspecting (might be wrong though) it might be related to this issue https://github.com/golang/go/issues/35407
Thanks for any help!
Contributor guide
Research direction
Start with the Activator throttling entry point in pkg/activator/net/throttler.go, especially breakerQueueDepth, and reproduce the reported load using a target service with maxReplicas=1 and containerConcurrency=2. Compare memory behavior under sustained backpressure and determine what bounded behavior would prevent the Activator from OOMing; done should include evidence that the reproduction no longer causes an OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100