knative / knative/serving

Activator will OOM under backpressure

Open
#13,583 8 comments 0 reactions 0 assignees View on GitHub
kind/bug triage/accepted
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

## What version of Knative?
0.22.3

## Expected Behavior

Activator should be resilient under backpressure

## Actual Behavior

Activator OOM'ed under backpressure

## Steps to Reproduce the Problem

1. Create an example target ksvc to mimic service with limited capacity (due to resource limitation, runtime/dependency bottleneck, or others). For the example here, I created a service with `maxReplicas=1, containerConcurrency=2`, and for each request it will sleep for 30s before returning a dummy response.
2. Attack the target ksvc with a throughput higher than it could handle. Example to attack with 200 rps with 1s timeout
```
ali --body 'dummy request body' -m POST --duration 7200s --rate 200 --timeout 1s https://ksvc-endpoint-url
```
3. Monitor activator resource usage: memory will heat up over time and eventually OOM. Increasing memory will only delay the OOM but not prevent it.

The default 600MB activator deployment OOM'ed after ~15 min.
image

Increased activator deployment memory to 2000MB. OOM'ed after ~50 min.
image

CPU utilization is low all through the period so activator is not scaled up (also not sure if scaling up activator can help here)

Is there any way/setting we can leverage to prevent activator from OOM under such backpressure, which could impact all ksvc in the cluster. Ideally if activator can deny surplus requests to the target ksvc if its queue is already long/full. Or if there's any tool we can borrow from Istio to enforce e.g. rate limiting on a per ksvc level? It looks like we do have a [breakerQueueDepth](https://github.com/knative/serving/blob/35abde024a891ba10565be82dfa167196324056d/pkg/activator/net/throttler.go#L56) but somehow it's not effective in terms of preventing OOM. Suspecting (might be wrong though) it might be related to this issue https://github.com/golang/go/issues/35407

Thanks for any help!

Contributor guide

Open the contributing guide

Research direction

Start with the Activator throttling entry point in pkg/activator/net/throttler.go, especially breakerQueueDepth, and reproduce the reported load using a target service with maxReplicas=1 and containerConcurrency=2. Compare memory behavior under sustained backpressure and determine what bounded behavior would prevent the Activator from OOMing; done should include evidence that the reproduction no longer causes an OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.