What is Kiwix/openZIM standard message broker?

This is kinda an old topic, but it looks like it might now be time to make a decision, create a plan and start implementing it.

This issue arises now because @elfkuzco came to the conclusion that it begins to be too complex to handle messaging between CMS components through DB and proposed to introduce a real messaging solution in https://github.com/openzim/cms/pull/201

This is an old topic already tracked in https://github.com/openzim/zimfarm/issues/1116 for the Zimfarm, and identified like "years ago".

What I've already identified in terms of requirements:

• throughput is expected to be very moderate, in the order of magnitude of few message per seconds
• we need persistence + backup
• we need at least once delivery guarantee ; up to the producer / consumer to achieve exactly once delivery if needed, typically with idempotent code
• we need a solution with acknowledgement ; consumer acknowledge the message has been received before the broker consider it as delivered
• we want this solution to be used only when the message consumer is a technical component and does not already exposes an API ; it will not replace traditional API communication
• we need some form of security to prevent unauthorized access to the messages

What we probably don't need:

• HA solution (like clustering, ...)
• a solution capable to cope with very big messages (we can ask apps to generate messages in the range of 1K or less, or rely on another protocol)
• a solution accessible by web interfaces (end-user UIs)

I assume we have some requirements linked to our k8s infra but it is very unclear to me. Do we have to require HTTPS (+Websocket) or can we accept any protocol (meaning we will expose it directly on the host port, like we already do for other services)?

In terms of plan, I propose that while we identify a good candidate solution, implement it in the CMS first and from there decide if we continue with this solution or change to another alternative. It could be a good moment to also decide if we deploy a central component or continue with one component per app.

@elfkuzco @rgaudin @kelson42 can you help me better identify requirements?

Start by reviewing the requirements in this issue and the related CMS pull request #201 and Zimfarm issue #1116. Compare candidate messaging solutions against delivery, persistence, security, protocol, and Kubernetes needs. Done means a broker is selected with an agreed deployment plan and a decision on central versus per-application components.