502 errors in cold start gatling tests caused by "Cannot initialize the action more than once"
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
## Environment details:
* Distributed test environment with 10 invokers
* latest drivers
## Steps to reproduce the issue:
1. perform the gatling cold start performance test:
https://github.com/apache/incubator-openwhisk/blob/master/tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala
2. parameters: 600s, 144 users
## Provide the expected results and outputs:
```
100% success rate, i.e. 200 response codes
```
## Provide the actual results and outputs:
```
~23/17500 = 0,13% failure rate with `502`
```
The 502 are application errors where the activations show the following error message:
```
"response": {
"statusCode": 2,
"result": {
"error": "Cannot initialize the action more than once."
}
},
```
## Additional information you deem important:
* issue happened 23x on 1 of the 10 invoker machines during a 3 min time window
* note that this test stresses the invoker systems and docker daemon at it's maximum
* docker daemon and VM might be slow at this time causing slow invoker and user container start ups
* monitoring shows that system is under hight load CPU wise during the test (kernel and user space)
* some issue (slow network, garbage collection, slow docker,...) caused the first `/init` to fail in invoker but succeed in the runtime (was not able to find the actual root-cause for that initial hickup)
* retry of invoker in `/init` failed with the exception shown above
* the failure is caused by a `403`, eg. https://github.com/apache/incubator-openwhisk-runtime-nodejs/commit/184f130c3e1d7c7ddda9fd3b0b6e677e262e9bfd#diff-4ca0b607ab76fb60662c0e52545ee30bR97
## Thoughts
Two options could be applicable here:
* the invoker could ignore the error and assume the container is initialised
* the runtimes could return a `200 - action already has been initialised` which would let the invoker proceed.
@markusthoemmes thanks for the input and discussion on the potential solutions!
@markusthoemmes @rabbah @tysonnorris @csantanapr thoughts?
Contributor guide
Research direction
Start with tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala and reproduce the cold-start test with 600s and 144 users. Then trace the invoker /init retry and the runtime's initialization response, using the reported 403 and "Cannot initialize the action more than once" error as markers. Done means the stress test achieves a 100% success rate with 200 responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, scala
- Domain
- distributed-systems, performance, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100