Separate the web UI from the build engine (buildbot multi-master)

Followup from the Europython discussions.

buildbot.python.org is one Twisted process that serves both the web UI and talks to all workers, so web load and build scheduling are competing. #755 demonstrates the issue with that, ~34 requests/second from crawlers, and it was OOM-killed twice, each time needing a manual restart. python/psf-salt#644 reduced the traffic but this proposal may help resolve the problem at its core I think.

Proposal

Use buildbot's multi-master support to run two masters against the shared database: an engine (workers, builders, schedulers, reporters, janitor) and a ui (web server, change hooks, dashboards, force scheduler).

The same-host split doesn't need DNS or load-balancer changes, buildbot.python.org already goes through HAProxy, and workers connect straight to the host via buildbot-api.python.org, which stays with the engine. And multi-master does not reduce database load but it multiplies the scheduling queries per engine, so two engines is I think the proper way on the current 2 vCPU / 4 GB Postgres (btw is it possible to upgrade the host?).

Risks

A webhook can be accepted and then never built as the split introduces this so if it's merged it must not go to production until it is fixed.

The hook runs on the ui master and the schedulers on the engine. If the engine is down when a webhook arrives:

• the ui stores the change and answers 200, so GitHub does not retry
• the event goes onto a transient queue with nobody listening, so it is dropped
• the engine never recovers it on restart, because schedulers only re-scan changes they had already classified

Today the hook and the schedulers share one process, so a restart gives GitHub an error and a retry instead of a silent loss.

The likely fix is to run the branch schedulers on the ui master next to the hook: buildrequests are recovered at startup, changes are not.

Losing the router can hang the masters. Buildbot's documentation says a master stops when it loses its connection to the router. Tested that scenario and the master does not stop, it closes its ports and marks itself inactive, but the process stays alive, so restarting on exit does not cover this situation.

Upstream calls multi-master experimental, the WAMP queue has no authentication so it needs to stay on loopback, and a possible split across hosts needs shared storage for /data/www/buildbot/test-results/.

A rough plan

1. master_role in master.cfg, defaulting to today's behaviour.
2. Local prototype against crossbar, which I've already done and it works.
3. psf-salt: crossbar, second basedir, Consul registration, supervision with a liveness probe, replacing the current make update-master and @reboot path.
4. Engine/ui split in production, once the webhook gap is resolved.

Points 3 and 4 will need psf-salt work.

Also the master should run under a systemd unit as now it starts from a @reboot cron entry, so when it was OOM-killed in #755 nothing restarted it. A unit with Restart= fixes could fix that but if the split happens, the unit also needs a liveness check that probes the port, because losing the router leaves the process alive but not serving.

Start with master.cfg and the proposed master_role, then review the local Crossbar prototype and the psf-salt work needed for a second basedir, Consul registration, supervision, and liveness probing. Verify webhook changes cannot be silently dropped when the engine is unavailable, and ensure the production split and systemd supervision recover both OOM and inactive-router cases.