Action stucks in error forever after 'too many requests' response occurred
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
I did performance test for **Openwhisk with New Scheduler** (pull-based activation). When it comes to test with 5000 RPS, hundred of error happened. Then, I checked it by invoked the tested action from WSK CLI and get error response
`error: Unable to invoke action 'perfTest': Too many requests (code 74bd570f6b048c437664907592f07bbe)`.
I waited for 4 hours with hope that the action will be normal again. But it seems like the action cannot be normal anymore (always return 'too many requests' error).
After long of waiting, I try to re-created the function (delete existing function and create the new one with the same name and kind) and still got `too many requests` error.
Is that the true behavior? or that is a bug?
## Environment details:
* distributed deployment, all VMs (AWS EC2 Instance) are Linux
* 2 VMs Controller, 4 VMs Invoker, 2 VMs Scheduler, 1 VMs ETCD
## Steps to reproduce:
1. Create simple nodejs action, for example `perfTest` action
2. Test `perfTest` action with 5000 RPS for 1 minutes. We use `wrk 2` as a tool. Below is the commad
`wrk -t10 -c10 -d60s -R5000 --script post.lua --latency https://owserver/api/v1/namespaces/_/actions/perfTest?blocking=false`
3. Wait until test finished with some of errors
4. Try to invoke `perfTest` action
## Expected results:
```
invocation success:
{
"params": {}
}
```
## Actual results:
```
invocation error:
error: Unable to invoke action 'perfTest': Too many requests (code 74bd570f6b048c437664907592f07bbe)
Run 'wsk --help' for usage.
```
## Additional information:
* The `perfTest`action back to normal after Scheduler was restarted. There us no other way besides that.
Contributor guide
Assessment
This issue has not been assessed yet.