Memory accounting, memory pool self-metrics and query memory limits for Prometheus

Proposal

Problem(s)

Presently, it's very difficult to reliably account for and control Prometheus's memory use on different tasks. The broad advice given for Prometheus operations usually boils down to "RAM is cheap, add more".

This is not practical in resource-constrained environments or where it is necessary to deploy Prometheus with numerous replicas (HA, sharded scrape, multi-tier, etc).

High memory use on one task can crash-restart the whole Prometheus instance

Since Prometheus has no support for accounting for or limiting memory use for various work, it's easy for e.g. a rogue query to OOM a Prometheus instance and disrupt scraping activity.

A database system like PostgreSQL can detect this and abort the query without disrupting other queries on the system. Prometheus has no such resilience.

Traditional workarounds for this when using Prometheus itself tend to multiply its already enormous memory use further, e.g. by running multiple agent-mode Prometheus instances to ensure scraping continuity, running a pool of HA Prometheus instances for resilience against crashes, etc.

Various 3rd party forks and variants try to work around this by splitting Prometheus into multiple cooperating processes, each of which may OOM without impacting the others. E.g. VictoricaMetrics, Thanos. This comes with high overheads of its own, and doesn't help when Prometheus itself forms a key piece of a lot of infrastructure.

Memory use is opaque and not exposed in metrics

Prometheus has no self-metrics tracking its memory use for different pools, so it's difficult to tell if memory pressure is due to:

• in-memory buffering of entire large scrape results
• scrape label rewrite cache (including memory used for dropping metrics)
• tsdb index maintenance (scrape/write) during WAL apply and replay
• tsdb index loading (queries)
• loading wide (high cardinality and/or long label pair values) label sets when executing queries
• queries loading lots of datapoints
• ... other

Prometheus does support golang's memory profiler tools (go tool prof) and exposes a HTTP API endpoint debug/pprof/heap to allow them to be run remotely. But this requires interactive, manual intervention, which is entirely counter to the continuous metrics model Prometheus itself advocates.

Proposal: memory accounting and quotas

Memory accounting metrics

Ideally Prometheus would report metrics akin to these (but with names created by someone smarter than me):

• prometheus_memory_scrape_label_rewrite_cache_size_bytes - gauge for current scrape label rewrite cache size
• prometheus_memory_scrape_buffers_size_bytes - gauge for current size of extant buffers holding raw scrape results
• prometheus_memory_scrape_tsdb_index_maintenance_size_bytes - amount of memory currently in-use for TSDB index maintenance when updating the inverted indexes for scrape results
• prometheus_memory_query_labelset_size_bytes - current memory used by label pairs loaded during query execution
• prometheus_memory_query_samples_size_bytes - current memory used for sample datapoints during query execution
• prometheus_memory_query_executor_working_set_size_bytes - memory currently consumed by query executor(s) to hold intermediate working sets of datapoints and labels during the execution of label-matching join operations, aggregations, subqueries, etc, excluding memory used by samples loaded from the TSDB and memory used by labelsets loaded from the TSDB indexes.
• ... etc

Cumulative counters for the same would also be maintained, to help keep track of allocator churn and provide insight into usage that slips between the gaps in in the scrape interval.

A configuration option could potentially toggle per-(job,instance) and potentially per-query dimensions for the above, at a higher runtime cost. Though it might make more sense to emit this info via OTLP tracing events only when required, given the likely cost of maintaining it.

Enforced memory quotas

The same memory accounting used to maintain memory-use metrics could be used to enforce per-scrape memory use limits and per-query memory use limits, protecting the stability of Prometheus as a whole.

Scrapes that reach their memory limit would be aborted, possibly in a partial and incomplete state where some scrape data has reached the tsdb and not other scrape data. A per-job-instance counter would be incremented to record the failure.

Queries that reach their memory limit would be aborted, and a HTTP 508 Resource Limit Reached response returned to the client.

Implementation

Both of these would require some form of heirachical memory tracking, since Prometheus would need to know things like

• memory used for scrapes
  • memory used for scrape label rewrite cache
    • for job A
    • ...
  • memory used for raw scrape results
    • for job A
    • ...
  • memory used for tsdb index maintenance when applying samples
    • for job A
    • ...
• memory used for query execution
  • memory used to hold label pairs loaded for queries
    • for query X
    • ...
  • memory used to hold series datapoints loaded for queries
    • for query X
    • ...
• ... and so on

Use a heirachical memory allocator

Unfortunately there is no mature, widely used heirachical allocator for golang that could be used as the basis for memory accounting and memory quotas in Prometheus, like PostgreSQL's palloc or Samba's talloc. This limits the available options, especially for enforcing per-query or per-scrape memory quotas.

This is unfortunate, because golang's context package would probably provide a good vehicle for carrying the execution context needed to enable a heirachical allocator to attribute memory to the correct pool/sub-pool.

Self-profile with golang runtime profiling

Golang has some capability to collect memory information already via runtime/pprof, runtime.MemStats, etc, and limited capabilities for memory labels or tags. It's likely that this could be used to self-profile Prometheus and expose metrics based on the resulting insights. The overhead of doing so would need to be measured and might be prohibitive, especially when used with a high tracking ratio for allocations and a high sample rate.

It's unlikely that the golang memory profiling features would be ideal for enforcing memory quotas, since they tend to lag one or two GC cycles behind current allocations and are maintained after-the-fact, but it might be enough to provide some basic capabilities. Prometheus could provide best-effort quota enforcement where queries with spiking memory use and/or high cost scrape jobs are flagged to request that they self-abort. Golang only has co-operative goroutine terminations anyway; the only way a goroutine can abort is if it checks a request-abort flag at various appropriate points and bail out.

Use golang memory arenas

The go 1.20 memory arenas feature had issues and is likely to be removed. Memory arenas could've helped Prometheus limit the memory a given query or scrape could use - though likely at the price of more wasted memory for small, simple queries.

Memory arenas won't help here.

Alternatives

I'm interested in others' thoughts on how such memory accounting and profiling might be achieved in Prometheus.

Start by reviewing the existing debug/pprof/heap endpoint and Go runtime interfaces mentioned in the proposal, including runtime/pprof, runtime.MemStats, and context. Define a scoped design for accounting and quota enforcement, then document measurable completion criteria for scrape and query memory metrics without assuming a particular allocator.