apache / apache/openwhisk

502 errors in cold start gatling tests caused by "Cannot initialize the action more than once"

Open
#3,939 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

## Environment details:

* Distributed test environment with 10 invokers
* latest drivers

## Steps to reproduce the issue:

1. perform the gatling cold start performance test:
https://github.com/apache/incubator-openwhisk/blob/master/tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala

2. parameters: 600s, 144 users

## Provide the expected results and outputs:

```
100% success rate, i.e. 200 response codes
```

## Provide the actual results and outputs:

```
~23/17500 = 0,13% failure rate with `502`
```

The 502 are application errors where the activations show the following error message:
```
"response": {
"statusCode": 2,
"result": {
"error": "Cannot initialize the action more than once."
}
},
```

## Additional information you deem important:
* issue happened 23x on 1 of the 10 invoker machines during a 3 min time window
* note that this test stresses the invoker systems and docker daemon at it's maximum
* docker daemon and VM might be slow at this time causing slow invoker and user container start ups
* monitoring shows that system is under hight load CPU wise during the test (kernel and user space)
* some issue (slow network, garbage collection, slow docker,...) caused the first `/init` to fail in invoker but succeed in the runtime (was not able to find the actual root-cause for that initial hickup)
* retry of invoker in `/init` failed with the exception shown above
* the failure is caused by a `403`, eg. https://github.com/apache/incubator-openwhisk-runtime-nodejs/commit/184f130c3e1d7c7ddda9fd3b0b6e677e262e9bfd#diff-4ca0b607ab76fb60662c0e52545ee30bR97

## Thoughts

Two options could be applicable here:
* the invoker could ignore the error and assume the container is initialised
* the runtimes could return a `200 - action already has been initialised` which would let the invoker proceed.

@markusthoemmes thanks for the input and discussion on the potential solutions!

@markusthoemmes @rabbah @tysonnorris @csantanapr thoughts?

Contributor guide

Open the contributing guide

Research direction

Start with tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala and reproduce the cold-start test with 600s and 144 users. Then trace the invoker /init retry and the runtime's initialization response, using the reported 403 and "Cannot initialize the action more than once" error as markers. Done means the stress test achieves a 100% success rate with 200 responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, scala
Domain
distributed-systems, performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.