GoogleCloudPlatform / GoogleCloudPlatform/ops-agent

Support config.d style directories for config merging

Open
#265 10 comments 2 reactions 0 assignees View on GitHub
no-stale
Dominant language
Go
Stars
204
Forks
87
Avg merge
21h 54m
Merged PRs (30d)
23

Description

My team is planning an Ops Agent rollout, where we will make use of the default system metrics and logging, including some custom metrics and custom logging rules, but also anticipate that downstream teams will their own custom metrics and logging rules.

This would require merging:

* Default configuration
* Platform team’s base configuration
* Custom metrics (organisation–wide)
* Custom logging (organisation–wide)
* Service-specific configuration
* Custom metrics (per–service)
* Custom logging (per–service)

Since different teams would be responsible for different parts of the configuration, the preferred way would be to have separate configuration files which are merged at runtime.

Fluent Bit (which Ops Agent uses) supports an [include syntax for configuration](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_include_file-1) which suits this use case well.

But I don't see an equivalent for Ops Agent, which expects a single configuration file for all rules contained within it.

Ideally, Ops Agent would support a config.d style directory, where each file is merged structurally. I expect this to be functionally similar to how the default config is merged in confgenerator/confmerger.go, except support an arbitrary number of inputs.

Until that's implemented, we need to look at workarounds:

* We use Puppet which has a [`concat`](https://forge.puppet.com/modules/puppetlabs/concat) module available, and multiple teams could have their configs concatenated, but YAML cannot safely be merged in a text–based fashion, due to differing whitespace.
* We could implement our own YAML–native merging prior to deployment (Hiera has this natively, or we could write our own)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.