lablup / lablup/backend.ai

Prometheus mmap file overflow causes IndexError cascade during manager shutdown

Open
#9,797 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

When running the manager in multiprocess mode, the prometheus_client mmap-backed .db file runs out of space, causing IndexError: mmap slice assignment is wrong size.

Root cause:

The Histogram metric backendai_layer_operation_duration_sec has 4 label dimensions (domain, layer, operation, success) and 9 custom buckets, creating ~13 time-series per label combination. The large cardinality from DomainType x LayerType x operation combinations exhausts the mmap file capacity.

The error cascades during shutdown because:

1. A shutdown operation (e.g. release_leadership, close, flush_database) triggers MetricPolicy to record metrics

2. The mmap file is full, so observe_layer_operation raises IndexError

3. This exception propagates to the except block in MetricPolicy.execute(), which also tries to record metrics

4. That also fails with the same mmap error, cascading through every dependency layer teardown

Affected code:

- src/ai/backend/common/metrics/metric.py:525-530 (Histogram definition)

- src/ai/backend/common/resilience/policies/metrics.py:91-108 (MetricPolicy error path)

Fix direction:

1. Wrap observe_layer_operation calls in MetricPolicy.execute() with try/except to suppress mmap errors and prevent the cascade

2. Reduce label cardinality - consider dropping the operation label from the Histogram, or collapsing domain+layer into fewer dimensions

JIRA Issue: BA-4948

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.