DiamondLightSource / DiamondLightSource/blueapi

Failed RabbitMQ Connection on Kubernetes not Handled

Open
#459 27 comments 0 reactions 0 assignees View on GitHub
bug helm chart needs confirmation
Dominant language
Python
Stars
13
Forks
13
Avg merge
2d 11h
Merged PRs (30d)
25

Description

#285 has made dev easier but broken prod!

Blueapi is capable of starting without a message bus for an easy dev experience, but on kubernetes, if the message bus connection fails, we want it to fall over so kubernetes restarts it.

Kubernetes cannot be instructed to start its pods in a certain order, it relies on eventual consistency. It throws up all pods simultaneously, if some depend on others they may fall over, kubernetes will restart them. It will repeat this process until all pods are happy with an exponential backoff.

## Proposed Solution

We could go back to falling over if there is config for a message bus but the connection fails, we could continue to not connect to a message bus if there is no config for one.

Thoughts @joeshannon @DiamondJoseph @ZohebShaikh

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.