the availablePermits of an invokerSlot may exceed the maxAllowed value when number of controller changed
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
## Environment details:
* native CentOS 7.3.1611 with docker 17.05-ce
* 4 controllers and 9 invokers, with invokerThreShold=16 and numCore * coreShare=20
## Steps to reproduce the issue:
1. invoke a single action in a infinitely loop in every one second
action code is:
```js
function main() {
sleep(50000)
return {payload: 'Hello world'};
}
function sleep(d){
for(var t = Date.now();Date.now() - t <= d;);
}
```
and the loop script is:
```shell
#!/bin/bash
while true
do
wsk -i action invoke hello --result
sleep 1s
done
```
as the `hello` action's execution time is about 50-51 seconds and it's invoked in every one second, there will be around 50 running containers in the cluster(make sure no other actions are invoked), which will make the home invoker(let's say it's invoker5) overloaded, while the cluster is still capable
2. stop a controller manually to simulate a failure and terminate the loop script in step 1
3. wait for all running actions finished, and then execute the loop script in step1 again
## Provide the expected results and outputs:
in the home invoker(invoker5), the total running containers should be:
```shell
# docker-runc list | grep running | wc -l
19
```
*19 = 16(running containers) + 2(prewarmed containers) + 1(invoker container)*
## Provide the actual results and outputs:
```
# docker-runc list | grep running | wc -l
23
```
*23 = 20(running containers) + 2(prewarmed containers) + 1(invoker container)*
## Explain
the reason is while cluster is changed, the `ShardingContainerPoolBalancer` will throw away the old `invokerSlots` and create a new sequence to replace it:
```scala
def updateCluster(newSize: Int): Unit = {
val actualSize = newSize max 1 // if a cluster size < 1 is reported, falls back to a size of 1 (alone)
if (_clusterSize != actualSize) {
_clusterSize = actualSize
val newTreshold = (totalInvokerThreshold / actualSize) max 1 // letting this fall below 1 doesn't make sense
currentInvokerThreshold = newTreshold
_invokerSlots = _invokerSlots.map(_ => new ForcableSemaphore(currentInvokerThreshold))
logging.info(
this,
s"loadbalancer cluster size changed to $actualSize active nodes. invokerThreshold = $currentInvokerThreshold")(
TransactionId.loadbalancer)
}
}
```
then the new states of all item in invokerSlots are all set to the max value(5=16/3 in this case), but there are still some invoking permits for some invokers not released(4 in there for the home invoker), when running actions are finished and release its permit, the `sync.permits` will increase and may exceed the `currentInvokerThreshold`(actually `sync.permits` can upto 5+4=9), which may cause the home invoker take more invoking requests than its capacity while some other invokers in idle state
Contributor guide
Assessment
This issue has not been assessed yet.