apache / apache/openwhisk

new invokers cause load balancer to reallocate black box invokers

Open
#3,693 1 comment 0 reactions 0 assignees View on GitHub
invoker
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

The logic to update the set of managed and black box invokers will cause new invokers (when a sufficient number is added) to become black box invokers, and reallocates ones that were black box invokers as managed invokers.

https://github.com/apache/incubator-openwhisk/blob/166189a8f15c99d9237e7020865a34c5bc92a0c2/core/controller/src/main/scala/whisk/core/loadBalancer/ShardingContainerPoolBalancer.scala#L422-L431

This can cause new invokers to re-pull images, and activations which might have been warn start on previous invokers now become cold starts.

Updating this test https://github.com/apache/incubator-openwhisk/blob/f64722498bc2f5eadc356b3ae26ebb85de1bcdbf/tests/src/test/scala/whisk/core/loadBalancer/test/ShardingContainerPoolBalancerTests.scala#L45

```
// apply third update, growing the state
val update3 = IndexedSeq(healthy(0), healthy(1), healthy(2))
state.updateInvokers(update3)

state.invokers shouldBe update3
state.managedInvokers shouldBe update3.dropRight(1)
state.blackboxInvokers shouldBe IndexedSeq(update3.last)
```

shows the change in invoker sets. The partitioning isn't quite right.

@markusthoemmes had a worthwhile idea in Slack while discussing this issue: briefly, an invoker comes up as a black box invoker explicitly and communicates this to the load balance). I'll leave it to him to document it further in a future comment.

Contributor guide

Open the contributing guide

Research direction

Start with ShardingContainerPoolBalancer.scala around lines 422-431 and the update3 case in tests/src/test/scala/whisk/core/loadBalancer/test/ShardingContainerPoolBalancerTests.scala. Run the load-balancer tests and inspect how state.updateInvokers partitions managed and black-box invokers. Done means new invokers no longer unnecessarily change existing black-box assignments, with a regression test covering the growing state.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.