apache / apache/openwhisk

active activation count for rate limiting is not affected on HTTP response

Open
#4,672 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

Creating a load test I received 429 rate limiting responses like `'' has exceeded its throttle limit, Too many concurrent requests in flight (count: 1600, allowed: 1600).`
This was unexpected because:
- the load test operates as 2000 virtual users
- the concurrentInvocations limit for the namespace is 8000

## Steps to reproduce the issue:

1. set namespace limit for concurrentActivations to 8000
2. run a gatling test with blocking requests and 2000 users
3. ? may be affected by using cosmosdb store, which can be slow to store activation records

## Provide the expected results and outputs:

Expect that since number of active requests should be 2000 (or less) at any moment, the 8000 limit should be fine.

## Provide the actual results and outputs:

I received many 429 responses indicating too many concurrent requests.

## Additional information you deem important:
* This may be related to using cosmos to store activations
* I expect similar would happen if log collection is slower than request arrival rate

I'm wondering what is the impact if we move the counter adjustment from`CommonLoadBalancer.processCompletion` into `CommonLoadBalancer.processResult`. From a user perspective, I expect the rate limiting to apply to requests that I have in flight, as opposed to requests that used to be in flight, but are still having logs collected or response records created.

Contributor guide

Open the contributing guide

Research direction

Start with CommonLoadBalancer.processCompletion and processResult, then reproduce the behavior with a Gatling test using blocking requests and 2,000 users. Compare the concurrent activation counter with the HTTP response lifecycle, including the possible effect of a slow Cosmos DB activation store. Done means in-flight requests are counted correctly and the test no longer produces premature 429 responses under the stated namespace limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.