apache / apache/openwhisk

the availablePermits of an invokerSlot may exceed the maxAllowed value when number of controller changed

Open
#3,655 1 comment 2 reactions 1 assignee Assigned to @cbickel View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

## Environment details:

* native CentOS 7.3.1611 with docker 17.05-ce
* 4 controllers and 9 invokers, with invokerThreShold=16 and numCore * coreShare=20

## Steps to reproduce the issue:

1. invoke a single action in a infinitely loop in every one second

action code is:
```js
function main() {
sleep(50000)
return {payload: 'Hello world'};
}

function sleep(d){
for(var t = Date.now();Date.now() - t <= d;);
}
```

and the loop script is:

```shell
#!/bin/bash
while true
do
wsk -i action invoke hello --result
sleep 1s
done
```

as the `hello` action's execution time is about 50-51 seconds and it's invoked in every one second, there will be around 50 running containers in the cluster(make sure no other actions are invoked), which will make the home invoker(let's say it's invoker5) overloaded, while the cluster is still capable

2. stop a controller manually to simulate a failure and terminate the loop script in step 1

3. wait for all running actions finished, and then execute the loop script in step1 again

## Provide the expected results and outputs:

in the home invoker(invoker5), the total running containers should be:

```shell
# docker-runc list | grep running | wc -l
19
```
*19 = 16(running containers) + 2(prewarmed containers) + 1(invoker container)*

## Provide the actual results and outputs:

```
# docker-runc list | grep running | wc -l
23
```
*23 = 20(running containers) + 2(prewarmed containers) + 1(invoker container)*

## Explain
the reason is while cluster is changed, the `ShardingContainerPoolBalancer` will throw away the old `invokerSlots` and create a new sequence to replace it:

```scala
def updateCluster(newSize: Int): Unit = {
val actualSize = newSize max 1 // if a cluster size < 1 is reported, falls back to a size of 1 (alone)
if (_clusterSize != actualSize) {
_clusterSize = actualSize
val newTreshold = (totalInvokerThreshold / actualSize) max 1 // letting this fall below 1 doesn't make sense
currentInvokerThreshold = newTreshold
_invokerSlots = _invokerSlots.map(_ => new ForcableSemaphore(currentInvokerThreshold))

logging.info(
this,
s"loadbalancer cluster size changed to $actualSize active nodes. invokerThreshold = $currentInvokerThreshold")(
TransactionId.loadbalancer)
}
}
```
then the new states of all item in invokerSlots are all set to the max value(5=16/3 in this case), but there are still some invoking permits for some invokers not released(4 in there for the home invoker), when running actions are finished and release its permit, the `sync.permits` will increase and may exceed the `currentInvokerThreshold`(actually `sync.permits` can upto 5+4=9), which may cause the home invoker take more invoking requests than its capacity while some other invokers in idle state

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.