v2.0.9 fatal error: runtime: out of memory (38 GB) with 2793 buckets (4262187 measurements) when using docker with 125 GB RAM

Steps to reproduce:

1. Provision a machine with 125 GB RAM (and 25 GB swap)
2. Create 2793 buckets (4262187 measurements) using 1.4 TB space of a 1.8 TB SSD (350 GB free space)
3. docker pull docker.io/influxdb:2.0.9 (similar issue with 2.0.8 and 2.0.7)
4. docker-compose up -d (pass --ui-disabled --pprof-disabled --session-renew-disabled --reporting-disabled --metrics-disabled)
5. InfluxDBv2 crashes during the startup time after about 10 seconds

Expected behavior:

I would expect InfluxDB detects when it cannot allocate more memory and handle that situation gracefully, especially during the startup time. I also expect that the health status of the container reveal that issue.

Actual behavior:

❌ 2.0.9 allocates more than 35% and crashes
❌ 2.0.8 allocates more than 35% and crashes
❌ 2.0.7 allocates less memory, but after filling more buckets has also crashed

Environment info:

• System info:
  • Linux 4.19.0-18-amd64 x86_64
  • Debian GNU/Linux 10 (buster)
• InfluxDB version: 2.0.7, 2.0.8, 2.0.9
• Other relevant environment details:
  • Docker version 20.10.9 (build c2ea9bc)
  • docker-compose version 1.29.2

Config:

Unmodified standard configuration using the official Docker image.

Extract of the docker-compose.yml:

  influx:
    image: docker.io/influxdb:2.0.9
    container_name: influx
    oom_kill_disable: true
    deploy:
      resources:
        limits:
          memory: 120g
        reservations:
          memory: 80g
    restart: unless-stopped
    user: 1000:1000
    command:
    - --reporting-disabled
    - --ui-disabled
    - --pprof-disabled
    - --session-renew-disabled
    - --metrics-disabled
    environment:
      INFLUXD_HTTP_IDLE_TIMEOUT: 10m0s
      TZ: UTC0
    expose:
    - 8086
    volumes:
    - /var/opt/influx:/var/lib/influxdb2:delegated
    - /var/opt/influx/etc:/etc/influxdb2:delegated
    - /etc/localtime:/etc/localtime:ro

Logs:

Logs 2.0.7

ts=2021-10-13T22:59:21.746737Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.746979Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=35725 error="[shard 35725] cannot allocate memory"
ts=2021-10-13T22:59:21.747536Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.748284Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=821 error="[shard 821] cannot allocate memory"
ts=2021-10-13T22:59:21.748370Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fc0a724b804543ee/autogen/22066 duration=296.180ms
ts=2021-10-13T22:59:21.749122Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=555 error="[shard 555] cannot allocate memory"
ts=2021-10-13T22:59:21.749888Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=37732 error="[shard 37732] cannot allocate memory"
ts=2021-10-13T22:59:21.750362Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=39741 error="[shard 39741] cannot allocate memory"
ts=2021-10-13T22:59:21.750469Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fc7e19aefc221ef6/autogen/217 duration=279.119ms
ts=2021-10-13T22:59:21.751367Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=1584 error="[shard 1584] cannot allocate memory"
ts=2021-10-13T22:59:21.751662Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=2905 error="[shard 2905] cannot allocate memory"
ts=2021-10-13T22:59:21.751697Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=1170 error="[shard 1170] cannot allocate memory"
ts=2021-10-13T22:59:21.752257Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.752874Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.753009Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.753658Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.754046Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=41749 error="[shard 41749] cannot allocate memory"
ts=2021-10-13T22:59:21.754156Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fbfe71db7bd7d7e2/autogen/31753 duration=379.844ms
ts=2021-10-13T22:59:21.754689Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=1984 error="[shard 1984] cannot allocate memory"
ts=2021-10-13T22:59:21.755457Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fca2147eadab2157/autogen/18566 duration=260.476ms
ts=2021-10-13T22:59:21.756043Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=1110 error="[shard 1110] cannot allocate memory"
ts=2021-10-13T22:59:21.756051Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=280 error="[shard 280] cannot allocate memory"
ts=2021-10-13T22:59:21.757190Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=8982 error="[shard 8982] cannot allocate memory"
ts=2021-10-13T22:59:21.757623Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.758480Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=16803 error="[shard 16803] cannot allocate memory"
ts=2021-10-13T22:59:21.759435Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=160 error="[shard 160] cannot allocate memory"
ts=2021-10-13T22:59:21.759651Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=98 error="[shard 98] cannot allocate memory"
ts=2021-10-13T22:59:21.759428Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=43916 error="[shard 43916] cannot allocate memory"
ts=2021-10-13T22:59:21.759444Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=4162 error="[shard 4162] cannot allocate memory"
ts=2021-10-13T22:59:21.759753Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=2860 error="[shard 2860] cannot allocate memory"
ts=2021-10-13T22:59:21.760198Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3144 error="[shard 3144] cannot allocate memory"
ts=2021-10-13T22:59:21.761090Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fcd030b5f0575e43/autogen/1149 duration=205.010ms
ts=2021-10-13T22:59:21.761944Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=502 error="[shard 502] cannot allocate memory"
ts=2021-10-13T22:59:21.762014Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fbfe71db7bd7d7e2/autogen/22227 duration=370.322ms
ts=2021-10-13T22:59:21.762119Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fce82f2816a1d63d/autogen/16619 duration=156.180ms
ts=2021-10-13T22:59:21.762153Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fca2147eadab2157/autogen/22010 duration=207.352ms
ts=2021-10-13T22:59:21.762174Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fce82f2816a1d63d/autogen/12115 duration=193.928ms
ts=2021-10-13T22:59:21.762171Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fce82f2816a1d63d/autogen/15111 duration=163.635ms
ts=2021-10-13T22:59:21.762426Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fc7e19aefc221ef6/autogen/106 duration=286.283ms
ts=2021-10-13T22:59:21.762944Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=41844 error="[shard 41844] cannot allocate memory"
ts=2021-10-13T22:59:21.763035Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=46014 error="[shard 46014] cannot allocate memory"
ts=2021-10-13T22:59:21.767292Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=41828 error="[shard 41828] cannot allocate memory"
ts=2021-10-13T22:59:21.767546Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=43873 error="[shard 43873] cannot allocate memory"
ts=2021-10-13T22:59:21.768164Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=45971 error="[shard 45971] cannot allocate memory"
ts=2021-10-13T22:59:21.774042Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=391 error="[shard 391] cannot allocate memory"
ts=2021-10-13T22:59:21.775267Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=7657 error="[shard 7657] cannot allocate memory"
ts=2021-10-13T22:59:21.775930Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=4283 error="[shard 4283] cannot allocate memory"
ts=2021-10-13T22:59:21.779508Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=8794 error="[shard 8794] cannot allocate memory"
ts=2021-10-13T22:59:21.780094Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=5387 error="[shard 5387] cannot allocate memory"
ts=2021-10-13T22:59:21.780237Z lvl=error msg="Failed to open shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=6512 error="[shard 6512] cannot allocate memory"
ts=2021-10-13T22:59:21.784433Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.785796Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fca2147eadab2157/autogen/16892 duration=257.919ms
ts=2021-10-13T22:59:21.790407Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.790881Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.791666Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fca2147eadab2157/autogen/20288 duration=245.278ms
ts=2021-10-13T22:59:21.791783Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fce82f2816a1d63d/autogen/13613 duration=195.115ms
ts=2021-10-13T22:59:21.795003Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.795930Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fce82f2816a1d63d/autogen/9134 duration=180.930ms
ts=2021-10-13T22:59:21.801319Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.803308Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fcf6ca11d0876ecf/autogen/20315 duration=183.894ms
ts=2021-10-13T22:59:21.805497Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.807542Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fcf6ca11d0876ecf/autogen/16905 duration=191.322ms
ts=2021-10-13T22:59:21.826457Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:21.828574Z lvl=info msg="Opened shard" log_id=0XAyvaIG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fcf6ca11d0876ecf/autogen/18593 duration=212.610ms
ts=2021-10-13T22:59:22.270382Z lvl=info msg="Reading file" log_id=0XAyvaIG000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/engine/wal/f964d121d4ebee6f/autogen/46150/_00002.wal size=4001122
ts=2021-10-13T22:59:22.295113Z lvl=info msg="index opened with 8 partitions" log_id=0XAyvaIG000 service=storage-engine index=tsi
ts=2021-10-13T22:59:35.692726Z lvl=info msg="Welcome to InfluxDB" log_id=0XAyyEZ0000 version=2.0.7 commit=2a45f0c037 build_date=2021-06-04T19:17:40Z

Last log line shows InfluxDB has been restarting. There is no panic logs providing some information about the crash. :-(

Logs 2.0.8

ts=2021-10-13T15:30:29.245523Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.245901Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.246326Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.246563Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd8722aad107aba6/autogen/13602 duration=32.062ms
ts=2021-10-13T15:30:29.247237Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.247446Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.248057Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.248113Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd4ff005000ebeab/autogen/5380 duration=71.208ms
ts=2021-10-13T15:30:29.248552Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fcf6ca11d0876ecf/autogen/22037 duration=82.282ms
ts=2021-10-13T15:30:29.248570Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd4ff005000ebeab/autogen/4255 duration=68.817ms
ts=2021-10-13T15:30:29.249361Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdaa3789af81d70b/autogen/584 duration=20.257ms
ts=2021-10-13T15:30:29.251084Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd8722aad107aba6/autogen/10606 duration=33.417ms
ts=2021-10-13T15:30:29.251149Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.251621Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd8722aad107aba6/autogen/15100 duration=33.647ms
ts=2021-10-13T15:30:29.252256Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd82ff1a1f048e1a/autogen/13588 duration=53.966ms
ts=2021-10-13T15:30:29.253733Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.253808Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd82ff1a1f048e1a/autogen/10592 duration=54.974ms
ts=2021-10-13T15:30:29.253989Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd82ff1a1f048e1a/autogen/9082 duration=41.208ms
ts=2021-10-13T15:30:29.254569Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.254927Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.255378Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=1110 error="[shard 1110] cannot allocate memory"
ts=2021-10-13T15:30:29.255707Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=4162 error="[shard 4162] cannot allocate memory"
ts=2021-10-13T15:30:29.255795Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3144 error="[shard 3144] cannot allocate memory"
ts=2021-10-13T15:30:29.255735Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd4ff005000ebeab/autogen/6505 duration=73.000ms
ts=2021-10-13T15:30:29.257714Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.258011Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.258089Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd8722aad107aba6/autogen/12104 duration=40.209ms
ts=2021-10-13T15:30:29.258068Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdaa3789af81d70b/autogen/1005 duration=31.395ms
ts=2021-10-13T15:30:29.258168Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=160 error="[shard 160] cannot allocate memory"
ts=2021-10-13T15:30:29.258159Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=280 error="[shard 280] cannot allocate memory"
ts=2021-10-13T15:30:29.258042Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=391 error="[shard 391] cannot allocate memory"
ts=2021-10-13T15:30:29.258036Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd8722aad107aba6/autogen/9092 duration=36.840ms
ts=2021-10-13T15:30:29.258017Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=502 error="[shard 502] cannot allocate memory"
ts=2021-10-13T15:30:29.258731Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.258789Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.260582Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.260753Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.261878Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.261904Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=41828 error="[shard 41828] cannot allocate memory"
ts=2021-10-13T15:30:29.262033Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=43873 error="[shard 43873] cannot allocate memory"
ts=2021-10-13T15:30:29.262060Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=45971 error="[shard 45971] cannot allocate memory"
ts=2021-10-13T15:30:29.262226Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.263242Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=6512 error="[shard 6512] cannot allocate memory"
ts=2021-10-13T15:30:29.255726Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=16803 error="[shard 16803] cannot allocate memory"
ts=2021-10-13T15:30:29.262589Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fe5288c979b21d10/autogen/35725 duration=17.835ms
ts=2021-10-13T15:30:29.262605Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdaa3789af81d70b/autogen/793 duration=30.866ms
ts=2021-10-13T15:30:29.262616Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd8722aad107aba6/autogen/16608 duration=42.398ms
ts=2021-10-13T15:30:29.262704Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=46014 error="[shard 46014] cannot allocate memory"
ts=2021-10-13T15:30:29.263404Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdb0bf6483dccc36/autogen/2927 duration=28.464ms
ts=2021-10-13T15:30:29.263425Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.263422Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.262718Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=41844 error="[shard 41844] cannot allocate memory"
ts=2021-10-13T15:30:29.262741Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=98 error="[shard 98] cannot allocate memory"
ts=2021-10-13T15:30:29.262757Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=43916 error="[shard 43916] cannot allocate memory"
ts=2021-10-13T15:30:29.263122Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.263897Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.264109Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fe5288c979b21d10/autogen/37732 duration=21.333ms
ts=2021-10-13T15:30:29.264114Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=5387 error="[shard 5387] cannot allocate memory"
ts=2021-10-13T15:30:29.264198Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fe5288c979b21d10/autogen/39741 duration=20.735ms
ts=2021-10-13T15:30:29.264327Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=7657 error="[shard 7657] cannot allocate memory"
ts=2021-10-13T15:30:29.264570Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=8794 error="[shard 8794] cannot allocate memory"
ts=2021-10-13T15:30:29.264575Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fe5288c979b21d10/autogen/41749 duration=17.961ms
ts=2021-10-13T15:30:29.265023Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.265110Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=4283 error="[shard 4283] cannot allocate memory"
ts=2021-10-13T15:30:29.265122Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.265300Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.266384Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.266399Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.266887Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.266935Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.267001Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/ff28cfff41e4ae71/autogen/8982 duration=14.719ms
ts=2021-10-13T15:30:29.267078Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.267480Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd82ff1a1f048e1a/autogen/16594 duration=55.314ms
ts=2021-10-13T15:30:29.268361Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd82ff1a1f048e1a/autogen/15086 duration=60.524ms
ts=2021-10-13T15:30:29.268420Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdda5d24ec0821c4/autogen/2860 duration=29.287ms
ts=2021-10-13T15:30:29.268703Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdb54a7c2dadd01c/autogen/821 duration=30.911ms
ts=2021-10-13T15:30:29.268998Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/ff023321b763d17b/autogen/1170 duration=20.401ms
ts=2021-10-13T15:30:29.269055Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fe75de0a5613d6eb/autogen/2905 duration=20.412ms
ts=2021-10-13T15:30:29.269348Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdb54a7c2dadd01c/autogen/1033 duration=33.532ms
ts=2021-10-13T15:30:29.269481Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fe1a0a6bb21ef93b/autogen/35728 duration=27.556ms
ts=2021-10-13T15:30:29.270039Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/ff023321b763d17b/autogen/1584 duration=20.610ms
ts=2021-10-13T15:30:29.270200Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fdb54a7c2dadd01c/autogen/555 duration=33.754ms
ts=2021-10-13T15:30:29.270243Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/ff023321b763d17b/autogen/1984 duration=19.081ms
ts=2021-10-13T15:30:29.271483Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd82ff1a1f048e1a/autogen/12090 duration=65.520ms
ts=2021-10-13T15:30:29.273582Z lvl=info msg="Opened file" log_id=0XA_F5MW000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/060cfba1d3e3cefc/autogen/29857/000000111-000000002.tsm id=0 duration=0.019ms
ts=2021-10-13T15:30:29.273608Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0XA_F5MW000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/060cfba1d3e3cefc/autogen/29857/000000111-000000002.tsm id=0 error="cannot allocate memory"
ts=2021-10-13T15:30:29.273690Z lvl=error msg="Failed to open shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open db_shard_id=29857 error="[shard 29857] cannot read corrupt file /var/lib/influxdb2/engine/data/060cfba1d3e3cefc/autogen/29857/000000111-000000002.tsm: cannot allocate memory"
ts=2021-10-13T15:30:29.323466Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/88589aaf789defe4/autogen/8782 duration=7635.659ms
ts=2021-10-13T15:30:29.389303Z lvl=info msg="index opened with 8 partitions" log_id=0XA_F5MW000 service=storage-engine index=tsi
ts=2021-10-13T15:30:29.468381Z lvl=info msg="Opened shard" log_id=0XA_F5MW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/88589aaf789defe4/autogen/6500 duration=7787.685ms
fatal error: runtime: cannot allocate memory
fatal error: runtime: cannot allocate memory
fatal error: runtime: cannot allocate memory
fatal error: runtime: cannot allocate memory
fatal error: runtime: cannot allocate memory
fatal error: runtime: cannot allocate memory
goroutine 18 [running]:
runtime.throw(0x2199065, 0x1f)
        /home/circleci/.tools/go/src/runtime/panic.go:1117 +0x74 fp=0xc0000c25e0 sp=0xc0000c25b0 pc=0x4410d4
runtime.newArenaMayUnlock(0x9d5a320)
        /home/circleci/.tools/go/src/runtime/mheap.go:2030 +0xdf fp=0xc0000c2618 sp=0xc0000c25e0 pc=0x4332ff
runtime.newMarkBits(0x1, 0x7f0501372040)
        /home/circleci/.tools/go/src/runtime/mheap.go:1950 +0xd8 fp=0xc0000c2660 sp=0xc0000c2618 pc=0x432e18
runtime.(*mspan).sweep(0x7f0322e2d978, 0x7f0322e2d900, 0x46fc00)
        /home/circleci/.tools/go/src/runtime/mgcsweep.go:471 +0x5a5 fp=0xc0000c2740 sp=0xc0000c2660 pc=0x42e5a5
runtime.sweepone(0x3ca9710)
        /home/circleci/.tools/go/src/runtime/mgcsweep.go:229 +0x28d fp=0xc0000c27a8 sp=0xc0000c2740 pc=0x42ddcd
runtime.bgsweep(0xc00010e000)
        /home/circleci/.tools/go/src/runtime/mgcsweep.go:166 +0xc5 fp=0xc0000c27d8 sp=0xc0000c27a8 pc=0x42da65
runtime.goexit()
        /home/circleci/.tools/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000c27e0 sp=0xc0000c27d8 pc=0x47bae1
created by runtime.gcenable
        /home/circleci/.tools/go/src/runtime/mgc.go:217 +0x5e
goroutine 1 [chan receive]:
github.com/influxdata/influxdb/v2/tsdb.(*Store).loadShards(0xc00000c3c0, 0x0, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/store.go:406 +0xa3b
github.com/influxdata/influxdb/v2/tsdb.(*Store).Open(0xc00000c3c0, 0x0, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/store.go:218 +0x25a
github.com/influxdata/influxdb/v2/storage.(*Engine).Open(0xc0004d2840, 0x3d2a038, 0xc000412c40, 0x0, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/storage/engine.go:175 +0xf6
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run(0xc0002fa8c0, 0x3d2a038, 0xc000412c40, 0xc000fe5200, 0x0, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:391 +0x2877
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1(0x0, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/cmd.go:106 +0x354
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc0004e38c0, 0xc0009ce9f0, 0x0, 0x3, 0x0, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/kit/cli/viper.go:54 +0x2b
github.com/spf13/cobra.(*Command).execute(0xc0004e38c0, 0xc000110050, 0x3, 0x3, 0xc0004e38c0, 0xc000110050)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc0004e38c0, 0xc001027f48, 0x1, 0x1)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
        /home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:53 +0x39b
goroutine 34 [chan receive]:
github.com/influxdata/flux/execute.createOperatorProfiler.func1(0xc000416620)
        /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.124.0/execute/profiler.go:108 +0xe5
created by github.com/influxdata/flux/execute.createOperatorProfiler
        /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.124.0/execute/profiler.go:106 +0xba
goroutine 14 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0002b03c0)
        /home/circleci/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0xcf
created by go.opencensus.io/stats/view.init.0
        /home/circleci/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x59
goroutine 85 [select]:
github.com/influxdata/influxdb/v2/kit/signals.WithSignals.func1(0xc000120d10, 0x3d2a038, 0xc000412c00, 0xc001e83da0)
        /home/circleci/go/src/github.com/influxdata/influxdb/kit/signals/context.go:18 +0xb1
created by github.com/influxdata/influxdb/v2/kit/signals.WithSignals
        /home/circleci/go/src/github.com/influxdata/influxdb/kit/signals/context.go:16 +0xcf
goroutine 9867 [chan receive]:
github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).saveWriter(0xc185b23600)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1787 +0x75
created by github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).SetMeasurementFieldSetWriter
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1762 +0xe8
goroutine 5805 [chan receive]:
github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).saveWriter(0xc01b0d69c0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1787 +0x75
created by github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).SetMeasurementFieldSetWriter
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1762 +0xe8
goroutine 67 [syscall]:
os/signal.signal_recv(0x0)
        /home/circleci/.tools/go/src/runtime/sigqueue.go:168 +0xa5
os/signal.loop()
        /home/circleci/.tools/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
        /home/circleci/.tools/go/src/os/signal/signal.go:151 +0x46
goroutine 86 [select]:
github.com/influxdata/influxdb/v2/bolt.(*pluginMetricsCollector).pollTelegrafStats(0xc000f46570, 0xc000206240)
        /home/circleci/go/src/github.com/influxdata/influxdb/bolt/metrics.go:109 +0x96
created by github.com/influxdata/influxdb/v2/bolt.(*pluginMetricsCollector).Open
        /home/circleci/go/src/github.com/influxdata/influxdb/bolt/metrics.go:104 +0x4b
goroutine 153 [chan receive]:
github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).saveWriter(0xc000b272c0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1787 +0x75
created by github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).SetMeasurementFieldSetWriter
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1762 +0xe8
goroutine 6755 [chan receive]:
github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).saveWriter(0xc0adb1d6c0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1787 +0x75
created by github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).SetMeasurementFieldSetWriter
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1762 +0xe8
goroutine 9108 [chan receive]:
github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).saveWriter(0xc1065602c0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1787 +0x75
created by github.com/influxdata/influxdb/v2/tsdb.(*MeasurementFieldSet).SetMeasurementFieldSetWriter
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1762 +0xe8

Logs 2.0.9

ts=2021-10-13T15:38:17.174218Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.177715Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a001fd01f6355be6/autogen/41814/000000002-00
ts=2021-10-13T15:38:17.177869Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a001fd01f6355be6/auto
ts=2021-10-13T15:38:17.178893Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.182402Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/9f4c4e9a1d95fb94/autogen/41689/000000053-00
ts=2021-10-13T15:38:17.182736Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a0524b7d282aa585/autogen/2901/000000001-000
ts=2021-10-13T15:38:17.183707Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.186654Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a0524b7d282aa585/auto
ts=2021-10-13T15:38:17.191027Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a059c9b37c2627a7/autogen/2169/000000244-000
ts=2021-10-13T15:38:17.192088Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.192197Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a059c9b37c2627a7/autogen/2169/000000244-000
ts=2021-10-13T15:38:17.192437Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a059c9b37c2627a7/auto
ts=2021-10-13T15:38:17.195729Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a001fd01f6355be6/autogen/46032/000000001-00
ts=2021-10-13T15:38:17.198118Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a001fd01f6355be6/auto
ts=2021-10-13T15:38:17.201982Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/9f4c4e9a1d95fb94/auto
ts=2021-10-13T15:38:17.204310Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.208366Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.211066Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a07c6bd133cee1be/autogen/8844/000000003-000
ts=2021-10-13T15:38:17.211185Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/autogen/5346/000000098-000
ts=2021-10-13T15:38:17.211276Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a07c6bd133cee1be/auto
ts=2021-10-13T15:38:17.215211Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.216669Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.218321Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a133252181e3fc90/autogen/1224/000000001-000
ts=2021-10-13T15:38:17.220661Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a133252181e3fc90/auto
ts=2021-10-13T15:38:17.225033Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.228631Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a0dd77d76439c168/autogen/2961/000000001-000
ts=2021-10-13T15:38:17.228792Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a0dd77d76439c168/auto
ts=2021-10-13T15:38:17.229270Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.232443Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a059c9b37c2627a7/autogen/2811/000000259-000
ts=2021-10-13T15:38:17.234156Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a059c9b37c2627a7/autogen/2811/000000259-000
ts=2021-10-13T15:38:17.234271Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a059c9b37c2627a7/auto
ts=2021-10-13T15:38:17.238992Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a133252181e3fc90/autogen/1977/000000001-000
ts=2021-10-13T15:38:17.241860Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/auto
ts=2021-10-13T15:38:17.249453Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a133252181e3fc90/auto
ts=2021-10-13T15:38:17.257094Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.257407Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/060cfba1d3e3cefc/auto
ts=2021-10-13T15:38:17.261533Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a0faa401b4ee8569/autogen/8993/000000001-000
ts=2021-10-13T15:38:17.261670Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a0faa401b4ee8569/auto
ts=2021-10-13T15:38:17.264778Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.271226Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/autogen/7610/000000135-000
ts=2021-10-13T15:38:17.271457Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/auto
ts=2021-10-13T15:38:17.276974Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.280929Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/autogen/6471/000000112-000
ts=2021-10-13T15:38:17.281124Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/auto
ts=2021-10-13T15:38:17.283874Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.287349Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.290023Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.293133Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/autogen/8747/000000032-000
ts=2021-10-13T15:38:17.293004Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a1672dc0b6796699/autogen/37697/000000004-00
ts=2021-10-13T15:38:17.293398Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a1672dc0b6796699/auto
ts=2021-10-13T15:38:17.294342Z lvl=info msg="Opened file" log_id=0XA_h1_G000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/autogen/4231/000000041-000
ts=2021-10-13T15:38:17.295519Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/auto
ts=2021-10-13T15:38:17.300000Z lvl=info msg="Opened shard" log_id=0XA_h1_G000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a14756f85dc0677c/auto
ts=2021-10-13T15:38:17.309449Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.309460Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.310476Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
ts=2021-10-13T15:38:17.311730Z lvl=info msg="index opened with 8 partitions" log_id=0XA_h1_G000 service=storage-engine index=tsi
fatal error: runtime: out of memory
runtime stack:
runtime.throw({0x1e3681c, 0x400000})
        /home/circleci/.tools/go/src/runtime/panic.go:1198 +0x71
runtime.sysMap(0xc96b000000, 0x1, 0x7f59ce0878a0)
        /home/circleci/.tools/go/src/runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x9b62420, 0x0)
        /home/circleci/.tools/go/src/runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x9b62420, 0x1, 0x0, 0x4)
        /home/circleci/.tools/go/src/runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
        /home/circleci/.tools/go/src/runtime/mheap.go:913 +0x69
runtime.systemstack()
        /home/circleci/.tools/go/src/runtime/asm_amd64.s:383 +0x46
goroutine 70304 [running]:
runtime.systemstack_switch()
        /home/circleci/.tools/go/src/runtime/asm_amd64.s:350 fp=0xc0024c4938 sp=0xc0024c4930 pc=0x46f6c0
runtime.(*mheap).alloc(0x7, 0xc0024c49b0, 0x9c, 0x1)
        /home/circleci/.tools/go/src/runtime/mheap.go:907 +0x73 fp=0xc0024c4988 sp=0xc0024c4938 pc=0x42e513
runtime.(*mcentral).grow(0xc0024c4a50)
        /home/circleci/.tools/go/src/runtime/mcentral.go:241 +0x65 fp=0xc0024c49d0 sp=0xc0024c4988 pc=0x41fe65
runtime.(*mcentral).cacheSpan(0x9b72968)
        /home/circleci/.tools/go/src/runtime/mcentral.go:161 +0x69e fp=0xc0024c4a48 sp=0xc0024c49d0 pc=0x41fc9e
runtime.(*mcache).refill(0x7f5a00cd0688, 0x4)
        /home/circleci/.tools/go/src/runtime/mcache.go:162 +0xaf fp=0xc0024c4a98 sp=0xc0024c4a48 pc=0x41eecf
runtime.(*mcache).nextFree(0x7f5a00cd0688, 0x4)
        /home/circleci/.tools/go/src/runtime/malloc.go:880 +0x85 fp=0xc0024c4ae0 sp=0xc0024c4a98 pc=0x414da5
runtime.mallocgc(0x10, 0x3508e20, 0x1)
        /home/circleci/.tools/go/src/runtime/malloc.go:1071 +0x4e8 fp=0xc0024c4b60 sp=0xc0024c4ae0 pc=0x415428
runtime.makeslice(0x1e0b773, 0x9, 0x0)
        /home/circleci/.tools/go/src/runtime/slice.go:98 +0x52 fp=0xc0024c4b88 sp=0xc0024c4b60 pc=0x4554f2
github.com/prometheus/client_golang/prometheus.NewDesc({0x1e0b773, 0x9}, {0x1e6d2a9, 0x23}, {0xc969de3f10, 0x1, 0x1}, 0x0)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.5.1/prometheus/desc.go:90 +0x166 fp=0xc0024c4e40 sp=0xc0024c4b88 pc=0xa280e6
github.com/prometheus/client_golang/prometheus.NewCounterVec({{0x0, 0x0}, {0x0, 0x0}, {0x1e0b773, 0x9}, {0x1e6d2a9, 0x23}, 0x0}, {0xc969de3f10, ...})
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.5.1/prometheus/counter.go:172 +0xc5 fp=0xc0024c4eb0 sp=0xc0024c4e40 pc=0xa27745
github.com/influxdata/influxdb/v2/pkg/rhh.NewMetrics({0x0, 0x0}, {0x0, 0x0}, 0x40f005)
        /home/circleci/go/src/github.com/influxdata/influxdb/pkg/rhh/metrics.go:89 +0xc45 fp=0xc0024c5348 sp=0xc0024c4eb0 pc=0xb66245
github.com/influxdata/influxdb/v2/pkg/rhh.NewHashMap({0x0, 0x50, 0x0, 0x0, 0x0})
        /home/circleci/go/src/github.com/influxdata/influxdb/pkg/rhh/rhh.go:34 +0x46 fp=0xc0024c5398 sp=0xc0024c5348 pc=0xb66706
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.NewTagBlockEncoder(...)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/tag_block.go:554
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).writeTagsetTo(0xc18e84bc20, {0x39296a0, 0xc336fa6900}, {0xc2370735f0, 0x9}, 0xc0024c56c8, 0xc0024c5620)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:929 +0xbb fp=0xc0024c5550 sp=0xc0024c5398 pc=0xc0133b
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).writeTagsetsTo(0xc336fa6900, {0x39296a0, 0xc336fa6900}, {0xc338cf4000, 0xbe15, 0x8}, 0x1e2118b, 0x0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:911 +0xac fp=0xc0024c55c0 sp=0xc0024c5550 pc=0xc011ec
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).CompactTo(0xc18e84bc20, {0x39300c0, 0xc1c39c5528}, 0x3, 0x3, 0xc3fa764120)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:841 +0x329 fp=0xc0024c5800 sp=0xc0024c55c0 pc=0xc009e9
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compactLogFile(0xc08086d8c0, 0xc18e84bc20)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:1220 +0x559 fp=0xc0024c5fb8 sp=0xc0024c5800 pc=0xc11139
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compact.func1()
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:978 +0x26 fp=0xc0024c5fe0 sp=0xc0024c5fb8 pc=0xc0e926
runtime.goexit()
        /home/circleci/.tools/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0024c5fe8 sp=0xc0024c5fe0 pc=0x471961
created by github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compact
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:977 +0x378
goroutine 1 [runnable]:
github.com/influxdata/influxdb/v2/tsdb.(*Store).loadShards(0xc00000c3c0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/store.go:405 +0x89d
github.com/influxdata/influxdb/v2/tsdb.(*Store).Open(0xc00000c3c0)
        /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/store.go:217 +0x216
github.com/influxdata/influxdb/v2/storage.(*Engine).Open(0xc0004049a0, {0x39711d8, 0xc00082a180})
        /home/circleci/go/src/github.com/influxdata/influxdb/storage/engine.go:178 +0x10f
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run(0xc0001d4c40, {0x39711d8, 0xc00082a140}, 0xc0004d8900)
        /home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:395 +0x2be2
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1()
        /home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/cmd.go:106 +0x2ec
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc00015c2c0, {0xc0007a71a0, 0x3, 0x3})
        /home/circleci/go/src/github.com/influxdata/influxdb/kit/cli/viper.go:54 +0x1e

Start by reproducing the startup failure with the provided docker-compose configuration and inspect storage-engine logs around tsdb_open and shard opening. Done means InfluxDB no longer crashes on this allocation failure and the container health status clearly reports the startup problem.