Prometheus mmap file overflow causes IndexError cascade during manager shutdown
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
When running the manager in multiprocess mode, the prometheus_client mmap-backed .db file runs out of space, causing IndexError: mmap slice assignment is wrong size.
Root cause:
The Histogram metric backendai_layer_operation_duration_sec has 4 label dimensions (domain, layer, operation, success) and 9 custom buckets, creating ~13 time-series per label combination. The large cardinality from DomainType x LayerType x operation combinations exhausts the mmap file capacity.
The error cascades during shutdown because:
1. A shutdown operation (e.g. release_leadership, close, flush_database) triggers MetricPolicy to record metrics
2. The mmap file is full, so observe_layer_operation raises IndexError
3. This exception propagates to the except block in MetricPolicy.execute(), which also tries to record metrics
4. That also fails with the same mmap error, cascading through every dependency layer teardown
Affected code:
- src/ai/backend/common/metrics/metric.py:525-530 (Histogram definition)
- src/ai/backend/common/resilience/policies/metrics.py:91-108 (MetricPolicy error path)
Fix direction:
1. Wrap observe_layer_operation calls in MetricPolicy.execute() with try/except to suppress mmap errors and prevent the cascade
2. Reduce label cardinality - consider dropping the operation label from the Histogram, or collapsing domain+layer into fewer dimensions
JIRA Issue: BA-4948
Contributor guide
Assessment
This issue has not been assessed yet.