[Proposal] Change Config Format

Problem

Single file alertmanager configuration with no external file loading creates a frustratingly hard system to configure and manage. This is evedenced further by the prometheus operator tickets here:

https://github.com/coreos/prometheus-operator/issues/2927,https://github.com/coreos/prometheus-operator/issues/2766, https://github.com/coreos/prometheus-operator/issues/1528, https://github.com/coreos/prometheus-operator/issues/1498, https://github.com/coreos/prometheus-operator/issues/2957

• Single file for config
• Inhibitions are global
• Routes, even root, can clobber sub routes if continue is set to false

Proposal

• Allow file loading of additional routes, receivers and inhibit rules using the currently implemented struct design
  • Routes loaded as files ( dashed lines below '- - -' ) would be appended to the root route loaded from the core config ('-------' lines below) within the child routes array

                   +-------> Child Route -------> Child Of Child Route
                   |
Route Route +------+-------> Child Route

                   +- - - -> Child Route

                   +- - - -> Child Route

• Config loading would protect against having a continue set to false for a root route, or a 1st generation child route
• Configs loaded as files would be done so in a non blocking manner. If parsing or validation fails then it would simply explain in the logs and not load it.

Specific Config changes

Core config file would have 3 additional sections to allow file loading elements. File loading would follow the current prometheus design firing up watchers for the paths given

routes_files
  - /etc/alertmanager/routes.yaml
receivers_files
  - /etc/alertmanager/receivers/*.yaml
inhibit_files
  - /etc/alertmanager/inhibit/*.yaml

Start by reviewing Alertmanager's existing configuration struct and current configuration-loading path, then compare how Prometheus handles file loading and watchers. Done means routes, receivers, and inhibition rules can be loaded from the proposed file patterns without blocking, invalid files are logged and skipped, and unsafe root or first-generation child routes are rejected.