influxdata / influxdata/influxdb
influxdb 2.0.7, 2.1.1 and 2.6.1 fail to start due to Out Of Memory (OOM) error
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. # systemctl start influxdb
__Expected behavior:__
Endpoint at port 8086.
__Actual behavior:__
OOM according to dmesg output:
```
[ 1333.857775] Call trace:
[ 1333.857786] dump_backtrace.part.0+0xec/0x100
[ 1333.857814] show_stack+0x20/0x30
[ 1333.857832] dump_stack_lvl+0x88/0xb4
[ 1333.857860] dump_stack+0x18/0x34
[ 1333.857880] dump_header+0x4c/0x210
[ 1333.857901] oom_kill_process+0x2bc/0x2c0
[ 1333.857920] out_of_memory+0xe8/0x354
[ 1333.857936] __alloc_pages+0xb04/0xea0
[ 1333.857953] __handle_mm_fault+0x9b4/0xa40
[ 1333.857976] handle_mm_fault+0x1b8/0x2d0
[ 1333.857997] do_page_fault+0x1f0/0x480
[ 1333.858021] do_translation_fault+0xa4/0xc0
[ 1333.858043] do_mem_abort+0x4c/0xa0
[ 1333.858071] el0_da+0x48/0xbc
[ 1333.858092] el0t_64_sync_handler+0xac/0x120
[ 1333.858116] el0t_64_sync+0x18c/0x190
[ 1333.858138] Mem-Info:
[ 1333.858158] active_anon:58628 inactive_anon:381223 isolated_anon:0
active_file:194 inactive_file:258 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:13880 slab_unreclaimable:6691
mapped:220 shmem:189 pagetables:1516
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:3513 free_pcp:121 free_cma:621
[ 1333.858227] Node 0 active_anon:234512kB inactive_anon:1524892kB active_file:976kB inactive_file:1032kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:880kB dirty:0kB writeback:0kB shmem:756kB writeback_tmp:0kB kernel_stack:2720kB pagetables:6064kB sec_pagetables:0kB all_unreclaimable? no
[ 1333.858282] DMA free:8040kB boost:0kB min:2540kB low:3412kB high:4284kB reserved_highatomic:0KB active_anon:137356kB inactive_anon:713244kB active_file:112kB inactive_file:240kB unevictable:0kB writepending:0kB present:970752kB managed:884684kB mlocked:0kB bounce:0kB free_pcp:296kB local_pcp:0kB free_cma:2484kB
[ 1333.858337] lowmem_reserve[]: 0 983 983 983
[ 1333.858403] DMA32 free:6012kB boost:7416kB min:10352kB low:11356kB high:12360kB reserved_highatomic:0KB active_anon:97156kB inactive_anon:811648kB active_file:636kB inactive_file:784kB unevictable:0kB writepending:0kB present:1048576kB managed:1007176kB mlocked:0kB bounce:0kB free_pcp:268kB local_pcp:0kB free_cma:0kB
[ 1333.858459] lowmem_reserve[]: 0 0 0 0
[ 1333.858523] DMA: 132*4kB (UEC) 98*8kB (UEC) 54*16kB (UEC) 26*32kB (UEC) 12*64kB (UEC) 3*128kB (UEC) 2*256kB (E) 1*512kB (C) 1*1024kB (U) 1*2048kB (U) 0*4096kB = 8256kB
[ 1333.858757] DMA32: 287*4kB (UME) 138*8kB (UME) 131*16kB (UME) 33*32kB (UE) 13*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6236kB
[ 1333.858945] 646 total pagecache pages
[ 1333.858963] 0 pages in swap cache
[ 1333.858981] Free swap = 0kB
[ 1333.858998] Total swap = 0kB
[ 1333.859016] 504832 pages RAM
[ 1333.859032] 0 pages HighMem/MovableOnly
[ 1333.859050] 31867 pages reserved
[ 1333.859066] 81920 pages cma reserved
[ 1333.859084] Tasks state (memory values in pages):
[ 1333.859101] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1333.859147] [ 193] 0 193 8381 271 86016 0 -250 systemd-journal
[ 1333.859181] [ 208] 0 208 4166 218 65536 0 0 systemd-userdbd
[ 1333.859214] [ 213] 0 213 7920 524 69632 0 -1000 systemd-udevd
[ 1333.859245] [ 218] 981 218 4639 301 69632 0 0 systemd-network
[ 1333.859284] [ 293] 980 293 5319 568 77824 0 0 systemd-resolve
[ 1333.859316] [ 294] 979 294 22623 234 73728 0 0 systemd-timesyn
[ 1333.859347] [ 299] 81 299 3453 270 65536 0 -900 dbus-daemon
[ 1333.859378] [ 300] 0 300 745 27 36864 0 0 dropbear
[ 1333.859408] [ 304] 0 304 4264 249 65536 0 0 systemd-logind
[ 1333.859439] [ 305] 0 305 4296 255 69632 0 0 systemd-homed
[ 1333.859470] [ 308] 0 308 572 20 36864 0 0 agetty
[ 1333.859501] [ 313] 0 313 2276 171 53248 0 0 dropbear
[ 1333.859531] [ 314] 0 314 1167 164 40960 0 0 bash
[ 1333.859562] [ 478] 0 478 2308 181 45056 0 0 dropbear
[ 1333.859591] [ 479] 0 479 1167 165 45056 0 0 bash
[ 1333.859624] [ 585] 976 585 3640612 434596 5009408 0 0 influxd
[ 1333.859655] [ 614] 0 614 4153 71 49152 0 0 journalctl
[ 1333.859685] [ 619] 0 619 4141 70 49152 0 0 systemd-userwor
[ 1333.859715] [ 620] 0 620 4141 70 49152 0 0 systemd-userwor
[ 1333.859744] [ 621] 0 621 4141 70 49152 0 0 systemd-userwor
[ 1333.859773] [ 622] 0 622 247 11 24576 0 0 dmesg
[ 1333.859802] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=influxd,pid=585,uid=976
[ 1333.860044] Out of memory: Killed process 585 (influxd) total-vm:14562448kB, anon-rss:1738384kB, file-rss:0kB, shmem-rss:0kB, UID:976 pgtables:4892kB oom_score_adj:0
```
According to journal:
```
Mar 06 18:09:16 influxPi4B influxd[623]: ts=2023-03-06T17:09:16.182008Z lvl=info msg="Opened file" log_id=0gPzDgI0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/.influxdbv2/engine/data/82c27d265e412abf/autogen/1016/000000001-000000001.tsm id=0 duration=6.467ms
Mar 06 18:09:16 influxPi4B influxd[623]: ts=2023-03-06T17:09:16.182617Z lvl=info msg="Reading file" log_id=0gPzDgI0000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/.influxdbv2/engine/wal/82c27d265e412abf/autogen/1016/_00001.wal size=1114353
```
__Environment info:__
* System info: Linux 6.1.14-1-rpi-ARCH aarch64
* InfluxDB version: InfluxDB v2.6.1 (git: 9dcf880fe0) build_date: 2022-12-29T15:52:06Z
* Other relevant environment details: disk info
```
Filesystem Size Used Avail Use% Mounted on
dev 758M 0 758M 0% /dev
run 924M 752K 924M 1% /run
/dev/mmcblk0p2 117G 22G 89G 20% /
tmpfs 924M 0 924M 0% /dev/shm
tmpfs 924M 0 924M 0% /tmp
/dev/mmcblk0p1 240M 55M 186M 23% /boot
```
__Config:__
From systemd .service file:
```
[Service]
CPUWeight=20
Nice=1
User=influxdb
Group=influxdb
LimitNOFILE=65536
ExecStart=/usr/bin/influxd --reporting-disabled --session-length=403200
KillMode=control-group
Restart=on-failure
```
__Reproduceable:__
Yes, always. I started with 2.0.7. Upgraded to 2.1.1 and finally to 2.6.1
Starting influxd result in an OOM with all 3 versions.
__Inspect:__
```
# influxd inspect verify-tsm --engine-path /var/lib/influxdb/.influxdbv2/engine
Broken Blocks: 0 / 1953097, in 289.689160683s
# influxd inspect verify-wal --wal-path /var/lib/influxdb/.influxdbv2/engine/wal
Results:
Files checked: 266
Total entries checked: 19856
Corrupt files found: None
Completed in 5.800122996s
```
__Performance:__
Impossible to generate profiles because the system becomes so slow that it is impossible to type any command after the OOM happens.
iostat output:
Linux 6.1.14-1-rpi-ARCH (influxPi4B) 03/06/23 _aarch64_ (4 CPU)
```
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 19.08 374.41 7.35 27.81 1.43 19.62 0.93 16.04 0.55 37.13 203.03 17.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 2.63
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 669.00 32860.00 48.00 6.69 4.44 49.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.97 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 762.38 25687.13 27.72 3.51 3.24 33.69 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.47 93.07
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 731.00 27440.00 47.00 6.04 3.46 37.54 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.53 98.40
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 1216.00 23508.00 91.00 6.96 2.74 19.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.33 97.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 1942.00 10488.00 143.00 6.86 1.70 5.40 2.00 40.00 8.00 80.00 4.00 20.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.30 93.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 882.00 22572.00 75.00 7.84 3.43 25.59 34.00 564.00 0.00 0.00 37.06 16.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.29 95.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 329.00 11416.00 80.00 19.56 11.33 34.70 137.00 3416.00 0.00 0.00 321.79 24.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 47.81 97.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 884.00 28372.00 305.00 25.65 5.78 32.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.11 99.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 1183.00 13236.00 30.00 2.47 0.84 11.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 85.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 716.00 15076.00 31.00 4.15 2.53 21.06 70.00 396.00 15.00 17.65 72.93 5.66 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.92 85.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 31.00 2488.00 0.00 0.00 322.29 80.26 125.00 10320.00 0.00 0.00 574.13 82.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 81.76 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 762.50 26411.54 408.65 34.89 12.16 34.64 15.38 3480.77 0.00 0.00 631.56 226.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18.99 103.85
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 797.12 35226.92 57.69 6.75 10.18 44.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.12 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 817.17 34537.37 151.52 15.64 10.37 42.26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.48 99.39
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 794.06 35164.36 227.72 22.29 10.07 44.28 1.98 51.49 10.89 84.62 18.50 26.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.04 100.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 752.38 36148.57 221.90 22.78 10.11 48.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.61 100.19
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 670.59 34345.10 150.98 18.38 11.00 51.22 8.82 1035.29 0.98 10.00 80.44 117.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.09 101.96
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 846.53 36427.72 56.44 6.25 11.88 43.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10.06 103.37
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 677.23 35421.78 72.28 9.64 12.57 52.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.51 97.03
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 738.83 37669.90 161.17 17.91 10.14 50.99 1.94 23.30 3.88 66.67 18.00 12.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.53 103.69
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 1122.86 49813.33 209.52 15.73 11.42 44.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.82 140.95
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 425.74 21013.86 62.38 12.78 11.02 49.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.69 58.22
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 1001.96 48823.53 493.14 32.98 11.12 48.73 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.15 135.29
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 640.59 28174.26 309.90 32.60 9.28 43.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.95 80.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 774.26 37200.00 413.86 34.83 9.24 48.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.15 103.76
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 626.73 27302.97 334.65 34.81 10.20 43.56 1.98 15.84 1.98 50.00 10.50 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.41 77.62
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 676.32 33803.51 331.58 32.90 14.51 49.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.81 93.68
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 779.55 39600.00 382.95 32.94 13.26 50.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10.34 110.45
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
mmcblk0 1312.71 78518.64 500.00 27.58 20.33 59.81 1.69 16.95 2.54 60.00 29.50 10.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 26.74 211.53
```
Contributor guide
Research direction
Start with the systemd service configuration and reproduce `systemctl start influxdb` on the reported Linux environment. Compare the `influxd` startup, OOM kernel output, and `inspect verify-tsm`/`verify-wal` results to identify the failing startup path. Done means a reproducible diagnosis and a verified fix or documented limitation for the affected versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- databases, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100