GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine

[Solution] rules-generated ConfigMap hits the 1MB etcd size limit when there are many Rules/ClusterRules/GlobalRules

Open
#1,923 0 comments 0 reactions 1 assignee Claimed by @gmp-bot View on GitHub
Dominant language
Go
Stars
232
Forks
109
Avg merge
1d 3h
Merged PRs (30d)
27

Description

## Problem

The operator packs all Rules, ClusterRules, and GlobalRules into a single `rules-generated` ConfigMap. Once you have enough rules (or rules with large content blobs), that ConfigMap exceeds the 1MB etcd object size limit and the operator starts failing with:

`ConfigMap "rules-generated" is invalid: []: Too long: must have at most 1048576 bytes`

Gzip compression helps delay this, but doesn't solve it. With enough rules you'll still hit the wall (which we unfortunately do).

## Expected behavior

The operator should be able to handle an arbitrary number of rules without running into etcd size limits. If a single ConfigMap isn't big enough, it should split the data across multiple ConfigMaps transparently.

## PR with a Suggested Solution

https://github.com/GoogleCloudPlatform/prometheus-engine/pull/1922

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.