new invokers cause load balancer to reallocate black box invokers
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
The logic to update the set of managed and black box invokers will cause new invokers (when a sufficient number is added) to become black box invokers, and reallocates ones that were black box invokers as managed invokers.
https://github.com/apache/incubator-openwhisk/blob/166189a8f15c99d9237e7020865a34c5bc92a0c2/core/controller/src/main/scala/whisk/core/loadBalancer/ShardingContainerPoolBalancer.scala#L422-L431
This can cause new invokers to re-pull images, and activations which might have been warn start on previous invokers now become cold starts.
Updating this test https://github.com/apache/incubator-openwhisk/blob/f64722498bc2f5eadc356b3ae26ebb85de1bcdbf/tests/src/test/scala/whisk/core/loadBalancer/test/ShardingContainerPoolBalancerTests.scala#L45
```
// apply third update, growing the state
val update3 = IndexedSeq(healthy(0), healthy(1), healthy(2))
state.updateInvokers(update3)
state.invokers shouldBe update3
state.managedInvokers shouldBe update3.dropRight(1)
state.blackboxInvokers shouldBe IndexedSeq(update3.last)
```
shows the change in invoker sets. The partitioning isn't quite right.
@markusthoemmes had a worthwhile idea in Slack while discussing this issue: briefly, an invoker comes up as a black box invoker explicitly and communicates this to the load balance). I'll leave it to him to document it further in a future comment.
Contributor guide
Research direction
Start with ShardingContainerPoolBalancer.scala around lines 422-431 and the update3 case in tests/src/test/scala/whisk/core/loadBalancer/test/ShardingContainerPoolBalancerTests.scala. Run the load-balancer tests and inspect how state.updateInvokers partitions managed and black-box invokers. Done means new invokers no longer unnecessarily change existing black-box assignments, with a regression test covering the growing state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100