Prometheus reads all .db files from PROMETHEUS_MULTIPROC_DIR without regard

prometheus-client==0.15.0

The method collect on prometheus_client.multiprocess.MultiProcessCollector reads all .db files, leading to memory issues, as it reads SQLite databases.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/........./........../.............py", line 20, in .........................
    data = generate_latest(registry)
  File "/usr/lib/python3.8/site-packages/prometheus_client/exposition.py", line 198, in generate_latest
    for metric in registry.collect():
  File "/usr/lib/python3.8/site-packages/prometheus_client/registry.py", line 97, in collect
    yield from collector.collect()
  File "/usr/lib/python3.8/site-packages/prometheus_client/multiprocess.py", line 153, in collect
    return self.merge(files, accumulate=True)
  File "/usr/lib/python3.8/site-packages/prometheus_client/multiprocess.py", line 45, in merge
    metrics = MultiProcessCollector._read_metrics(files)
  File "/usr/lib/python3.8/site-packages/prometheus_client/multiprocess.py", line 73, in _read_metrics
    for key, value, _ in file_values:
  File "/usr/lib/python3.8/site-packages/prometheus_client/mmap_dict.py", line 43, in _read_all_values
    value = _unpack_double(data, pos)[0]
struct.error: unpack_from requires a buffer of at least 1634562696 bytes for unpacking 8 bytes at offset 1634562688 (actual buffer size is 12288)

The issue is on collect() method:

def collect(self):
    files = glob.glob(os.path.join(self._path, '*.db'))
    return self.merge(files, accumulate=True)

I believe that there should be some kind of file validation to assure that the file is related to a prometheus metric, e.g:

import re

POSSIBLE_PROMETHEUS_FILENAMES_INCLUDE = {'counter', '...', 'histogram', 'gauge_livesum'}

def collect(self):
    files = glob.glob(os.path.join(self._path, '*.db'))
    #
    # assert files
    valid_files =  [f for f in files if re.split('_\d+.db', f)[0] in POSSIBLE_PROMETHEUS_FILENAMES_INCLUDE]
    #
    #
    return self.merge(valid_files, accumulate=True)

Commencez dans prometheus_client/multiprocess.py, au niveau de MultiProcessCollector.collect, et suivez son appel à merge et mmap_dict.py tout en reproduisant la traceback signalée de prometheus-client 0.15.0. Déterminez comment les fichiers de PROMETHEUS_MULTIPROC_DIR doivent être reconnus, puis vérifiez que les fichiers .db sans rapport ou mal formés n'entraînent plus la lecture de données invalides lors de la collecte, tandis que les fichiers de métriques valides continuent de fonctionner.