Support control messages via MQ

This ticket is a migrated Trac ticket [3203](http://trac.buildbot.net/ticket/3203)

People contributed to the original ticket: @djmitche, @tardyp
Ticket created on: `Feb 14 2015`
Ticket last modified on: `Dec 22 2015`

---

For

* #1039 - triggerable
* #3202 - force
* #2639 - rebuild
* #3156 - stop

we need a way to communicate back and forth between the frontend web UI and backend schedulers (and other components).

This might be some kind of IPC over MQ.

---

_Comment from_: @dustin
_Date_: `Sep 06 2015`

I suspect some of this has been implemented -- @tardyp?

---
_Comment from_: @tardyp
_Date_: `Sep 06 2015`

only stop is implemented via mq, because it is mandatory.

rebuild, triggereable, and force are implemented on the master that execute the control rest api (as of today)

Note that the goal of everything via mq is to allow for a www-only master to have a read-only creds to the db.

---
_Comment from_: @dustin
_Date_: `Sep 06 2015`

For a 0.9.0 release, i think we need to have an understandable story about multi-master support. I'd really rather that not be "doesn't work" (since we have had decent multi-master support in 8).

Is there some simply way to describe what does and does not work right now?

---
_Comment from_: @tardyp
_Date_: `Dec 22 2015`

A simple way to say it is by adding the concept of symetrics and asymetrics multimaster (like there is SMP and AMP for multi core CPUs)

Symetric multimaster is when each master share the exact same configuration. They run the same builders, same schedulers, same everything, the only difference is that slaves are connected evenly between the masters (by any means (e.g. DNS loadbalancing, etc))
Symetric multimaster is good to use to scale buildbot horizontally.

Asymetric multimaster is when each master have different configuration. Each master may have a specific responsability (e.g schedulers, set of builder, UI). This was more how you did in 0.8, also because of its own technical limitations.
A nice feature of asymetric multimaster is that you can have the UI only handled by some masters.

Separating the UI from the controlling will greatly help in the performance of the UI, because badly written [[BuildSteps]] can stall the reactor for several seconds.

asymetric multimaster is a more difficult goal to achieve, even if I understand the need, this should not block nine release

---
_Comment from_: @dustin
_Date_: `Dec 22 2015`

OK, so symmetric multimaster (SMM) is working now, and AMM is a 0.9.+ feature.