apache / apache/openwhisk

Action stucks in error forever after 'too many requests' response occurred

Open
#5,368 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

I did performance test for **Openwhisk with New Scheduler** (pull-based activation). When it comes to test with 5000 RPS, hundred of error happened. Then, I checked it by invoked the tested action from WSK CLI and get error response

`error: Unable to invoke action 'perfTest': Too many requests (code 74bd570f6b048c437664907592f07bbe)`.

I waited for 4 hours with hope that the action will be normal again. But it seems like the action cannot be normal anymore (always return 'too many requests' error).

After long of waiting, I try to re-created the function (delete existing function and create the new one with the same name and kind) and still got `too many requests` error.

Is that the true behavior? or that is a bug?

## Environment details:

* distributed deployment, all VMs (AWS EC2 Instance) are Linux
* 2 VMs Controller, 4 VMs Invoker, 2 VMs Scheduler, 1 VMs ETCD

## Steps to reproduce:
1. Create simple nodejs action, for example `perfTest` action
2. Test `perfTest` action with 5000 RPS for 1 minutes. We use `wrk 2` as a tool. Below is the commad
`wrk -t10 -c10 -d60s -R5000 --script post.lua --latency https://owserver/api/v1/namespaces/_/actions/perfTest?blocking=false`
3. Wait until test finished with some of errors
4. Try to invoke `perfTest` action

## Expected results:

```
invocation success:

{
"params": {}
}
```

## Actual results:

```
invocation error:

error: Unable to invoke action 'perfTest': Too many requests (code 74bd570f6b048c437664907592f07bbe)
Run 'wsk --help' for usage.
```

## Additional information:
* The `perfTest`action back to normal after Scheduler was restarted. There us no other way besides that.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.