GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine
[Solution] rules-generated ConfigMap hits the 1MB etcd size limit when there are many Rules/ClusterRules/GlobalRules
- Dominant language
- Go
- Stars
- 232
- Forks
- 109
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 27
Description
## Problem
The operator packs all Rules, ClusterRules, and GlobalRules into a single `rules-generated` ConfigMap. Once you have enough rules (or rules with large content blobs), that ConfigMap exceeds the 1MB etcd object size limit and the operator starts failing with:
`ConfigMap "rules-generated" is invalid: []: Too long: must have at most 1048576 bytes`
Gzip compression helps delay this, but doesn't solve it. With enough rules you'll still hit the wall (which we unfortunately do).
## Expected behavior
The operator should be able to handle an arbitrary number of rules without running into etcd size limits. If a single ConfigMap isn't big enough, it should split the data across multiple ConfigMaps transparently.
## PR with a Suggested Solution
https://github.com/GoogleCloudPlatform/prometheus-engine/pull/1922
Contributor guide
Assessment
This issue has not been assessed yet.