apache / apache/openwhisk

Round-robin HA controller can lead to concurrent modications errors

Open
#3,032 6 comments 0 reactions 0 assignees View on GitHub
bug controller
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

Since Nginx does proxy in Round-Robin way, some tests which were designed to run sequentially in one controller are failed.

Let me explain one example, travis of latest commit(0737826c06c73a904c49977910d103d1eed79174) failed the test "Wsk Package should allow updating a package". Because the result of package update was 409, not 200

See the code below
```
it should "allow updating a package" in withAssetCleaner(wskprops) { (wp, assetHelper) =>
val name = "simplepackagetoupdate"
assetHelper.withCleaner(wsk.pkg, name) { (pkg, _) =>
pkg.create(name, params1)
pkg.create(name, params2, update = true)
}
}
```

It might runs in this way
```
1. delete pkg when withCleaner starts -> controller0 delete package if it exists
2. pkg.create(name, params1) -> controller1 put package with rev A
3. pkg.create(name, params1) -> controller0 put package with rev B
4. delete pkg when withCleaner ends -> controller1 del package with rev A
```
Package is finally stored with rev B, but controller1 try to delete it with rev A, and then CouchDB response last operation as a conflict(409)
It could be success, if 'cacheInvalidation' msg arrived between 3 & 4. But we can't guarantee that.

I found that many tests were failed by similar scenario while running Travis.

Any good idea?

[tarvis log](https://travis-ci.org/apache/incubator-openwhisk/builds/309179905?utm_source=github_status&utm_medium=notification)
[controller0_logs.log](https://github.com/apache/incubator-openwhisk/files/1516179/controller0_logs.log)
[controller1_logs.log](https://github.com/apache/incubator-openwhisk/files/1516250/controller1_logs.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.