Controller should not assume DB is initialized when it starts
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
This issue came up recently and I'm opening up the discussion here so that we take a decision if we want to fix this or not. I think that in a distributed environment, with a system that depends on other components, making assumptions about the state of other components is not going to work all the time.
Such is the case with the Controller and the DB. In a recent PR we introduced a nice way for the controller to health-check invokers by asking them to execute a JS action. This only works if the DB is initialized before the controller starts. If that's not the case then the controller is like in a dead-lock mode; it tries to ping invokers, invokers complain they can't fetch the action, and in this time the controller doesn't verify that the action to be executed exists in the DB.
As a result we could have healthy invokers, healthy controllers, but all invocations fail b/c the controller believes there are no healthy invokers available.
A good way to test this is to spin up [docker-compose](https://github.com/apache/incubator-openwhisk-devtools) . `docker-compose` starts the Controller and the DB about the same time, but the DB takes longer to be initialized, and by the time the ansible script finished, it is already too late for the controller. But we could think of other cases when the DB has the action initially only, but after a while, due to a change, the action `invokerHealthTestAction` is mistakenly deleted, and we're back in the same situation.
WDYT ?
Contributor guide
Assessment
This issue has not been assessed yet.