apache / apache/openwhisk

LoadBalancerService does not consider concurrent requests when scheduling

Open
#2,859 4 comments 1 reaction 0 assignees View on GitHub
controller invoker
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

Under concurrent requests to controller, LoadBalancerService will not consider concurrent requests when scheduling to an invoker.

For example, consider `LoadBalacnerServiceObjectTests.scala`, with the following test
```
it should "consider load immediately and jump to next invoker when overloaded" in {
val invokerCount = 10
val hash = 2
val targetInvoker = hash % invokerCount

val invs = invokers(invokerCount).updated(targetInvoker, (InstanceId(targetInvoker), Healthy, 0))
val step = hashInto(LoadBalancerService.pairwiseCoprimeNumbersUntil(invokerCount), hash)

LoadBalancerService.schedule(invs, 1, hash) shouldBe Some(InstanceId((hash + 0) % invs.size))
LoadBalancerService.schedule(invs, 1, hash) shouldBe Some(InstanceId((hash + 1) % invs.size))

}
```

We have:
* set the existing load to 0 to start with
* set the invokerBusyThreshold to 1 on `scheduler()` invocations
* schedule one activation with hash=2
* schedule second activation with hash=2

In this case, but activations are scheduled to InstanceId(2), but the second should be scheduled to next invoker since busy threshold is exceeded.

I believe this is related to the changing of `LoadbalancerData.activationCountPerInvoker()` from sync to async. Since LoadBalancerService is processed with multiple threads, the updates to the LoadBalancerData are not seen immediately by other threads.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.