influxdata / influxdata/influxdb
2.0.7 Compacting file result in OOM on Raspberry Pi 4B
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
1. Disable swap
2. Insert new measurement every second using `socat` to `/api/v2/write` via python script
3. Wait 23 days
__Expected behavior:__
No OOM
__Actual behavior:__
influxd OOM
__Environment info:__
* Linux 5.10.48-1-ARCH aarch64
* InfluxDB version: InfluxDB 2.0.7 (git: 2a45f0c037) build_date: 2021-06-04T19:17:40Z
* Disk usage:
```Filesystem Size Used Avail Use% Mounted on
dev 665M 0 665M 0% /dev
run 925M 744K 925M 1% /run
/dev/mmcblk0p2 117G 11G 101G 10% /
tmpfs 925M 0 925M 0% /dev/shm
tmpfs 925M 0 925M 0% /tmp
/dev/mmcblk0p1 240M 43M 198M 18% /boot
tmpfs 185M 0 185M 0% /run/user/0
```
__Logs:__
```
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357097Z lvl=info msg="TSM compaction (start)" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy>
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357871Z lvl=info msg="Beginning compaction" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=l>
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357929Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357967Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358003Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358039Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358074Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358113Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358148Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358182Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:27 influxPi4B kernel: influxd invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Jul 31 11:01:27 influxPi4B kernel: CPU: 3 PID: 17748 Comm: influxd Tainted: G C 5.10.48-1-ARCH #1
Jul 31 11:01:27 influxPi4B kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
Jul 31 11:01:27 influxPi4B kernel: Call trace:
Jul 31 11:01:27 influxPi4B kernel: dump_backtrace+0x0/0x1a0
Jul 31 11:01:27 influxPi4B kernel: show_stack+0x18/0x24
Jul 31 11:01:27 influxPi4B kernel: dump_stack+0xc8/0x120
Jul 31 11:01:27 influxPi4B kernel: dump_header+0x48/0x1f4
Jul 31 11:01:27 influxPi4B kernel: oom_kill_process+0x204/0x20c
Jul 31 11:01:27 influxPi4B kernel: out_of_memory+0xe4/0x320
Jul 31 11:01:27 influxPi4B kernel: __alloc_pages_slowpath.constprop.0+0x850/0xba0
Jul 31 11:01:27 influxPi4B kernel: __alloc_pages_nodemask+0x298/0x2f0
Jul 31 11:01:27 influxPi4B kernel: handle_mm_fault+0x4dc/0xc90
Jul 31 11:01:27 influxPi4B kernel: do_page_fault+0x140/0x3d4
Jul 31 11:01:27 influxPi4B kernel: do_translation_fault+0xb0/0xcc
Jul 31 11:01:27 influxPi4B kernel: do_mem_abort+0x44/0xa4
Jul 31 11:01:27 influxPi4B kernel: el0_da+0x28/0x34
Jul 31 11:01:27 influxPi4B kernel: el0_sync_handler+0x168/0x1b0
Jul 31 11:01:27 influxPi4B kernel: el0_sync+0x180/0x1c0
Jul 31 11:01:27 influxPi4B kernel: Mem-Info:
Jul 31 11:01:27 influxPi4B kernel: active_anon:206675 inactive_anon:243243 isolated_anon:0
active_file:239 inactive_file:308 isolated_file:32
unevictable:0 dirty:0 writeback:0
slab_reclaimable:4988 slab_unreclaimable:5648
mapped:442 shmem:179 pagetables:1405 bounce:0
free:3532 free_pcp:31 free_cma:215
Jul 31 11:01:27 influxPi4B kernel: Node 0 active_anon:826700kB inactive_anon:972972kB active_file:956kB inactive_file:1232kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1768>
Jul 31 11:01:27 influxPi4B kernel: DMA free:8144kB min:6648kB low:7524kB high:8400kB reserved_highatomic:0KB active_anon:529016kB inactive_anon:332500kB active_file:200kB inactive_file:204kB u>
Jul 31 11:01:27 influxPi4B kernel: lowmem_reserve[]: 0 984 984 984
Jul 31 11:01:27 influxPi4B kernel: DMA32 free:5984kB min:10340kB low:11344kB high:12348kB reserved_highatomic:0KB active_anon:297424kB inactive_anon:640448kB active_file:876kB inactive_file:14>
Jul 31 11:01:27 influxPi4B kernel: lowmem_reserve[]: 0 0 0 0
Jul 31 11:01:27 influxPi4B kernel: DMA: 358*4kB (UMEC) 241*8kB (UMEC) 129*16kB (UMEC) 59*32kB (UEC) 16*64kB (UEC) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8336kB
Jul 31 11:01:27 influxPi4B kernel: DMA32: 364*4kB (UM) 161*8kB (UME) 174*16kB (UE) 24*32kB (UME) 2*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6424kB
Jul 31 11:01:27 influxPi4B kernel: 725 total pagecache pages
Jul 31 11:01:27 influxPi4B kernel: 0 pages in swap cache
Jul 31 11:01:27 influxPi4B kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 31 11:01:27 influxPi4B kernel: Free swap = 0kB
Jul 31 11:01:27 influxPi4B kernel: Total swap = 0kB
Jul 31 11:01:27 influxPi4B kernel: 504832 pages RAM
Jul 31 11:01:27 influxPi4B kernel: 0 pages HighMem/MovableOnly
Jul 31 11:01:27 influxPi4B kernel: 31338 pages reserved
Jul 31 11:01:27 influxPi4B kernel: 65536 pages cma reserved
Jul 31 11:01:27 influxPi4B kernel: Tasks state (memory values in pages):
Jul 31 11:01:27 influxPi4B kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jul 31 11:01:27 influxPi4B kernel: [ 225] 0 225 3952 190 61440 0 0 systemd-userdbd
Jul 31 11:01:27 influxPi4B kernel: [ 229] 0 229 4366 403 73728 0 -1000 systemd-udevd
Jul 31 11:01:27 influxPi4B kernel: [ 233] 981 233 5397 264 73728 0 0 systemd-network
Jul 31 11:01:27 influxPi4B kernel: [ 311] 980 311 4973 481 69632 0 0 systemd-resolve
Jul 31 11:01:27 influxPi4B kernel: [ 312] 979 312 22392 205 65536 0 0 systemd-timesyn
Jul 31 11:01:27 influxPi4B kernel: [ 315] 81 315 3554 337 57344 0 -900 dbus-daemon
Jul 31 11:01:27 influxPi4B kernel: [ 316] 0 316 1880 134 49152 0 0 socat
Jul 31 11:01:27 influxPi4B kernel: [ 318] 0 318 4067 225 57344 0 0 systemd-homed
Jul 31 11:01:27 influxPi4B kernel: [ 320] 0 320 4088 245 65536 0 0 systemd-logind
Jul 31 11:01:27 influxPi4B kernel: [ 322] 0 322 99907 5174 131072 0 0 python
Jul 31 11:01:27 influxPi4B kernel: [ 323] 0 323 2403 198 49152 0 -1000 sshd
Jul 31 11:01:27 influxPi4B kernel: [ 325] 0 325 2116 129 45056 0 0 agetty
Jul 31 11:01:27 influxPi4B kernel: [ 17654] 976 17654 2059639 437575 4632576 0 0 influxd
Jul 31 11:01:27 influxPi4B kernel: [ 17658] 0 17658 8115 227 69632 0 -250 systemd-journal
Jul 31 11:01:27 influxPi4B kernel: [ 38555] 0 38555 4079 188 61440 0 0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: [ 38556] 0 38556 4079 187 65536 0 0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: [ 38557] 0 38557 4079 188 57344 0 0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=influxd,pid=17654,uid=976
Jul 31 11:01:27 influxPi4B kernel: Out of memory: Killed process 17654 (influxd) total-vm:8238556kB, anon-rss:1749696kB, file-rss:604kB, shmem-rss:0kB, UID:976 pgtables:4524kB oom_score_adj:0
Jul 31 11:01:27 influxPi4B kernel: oom_reaper: reaped process 17654 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 31 11:01:27 influxPi4B systemd[1]: influxdb.service: Main process exited, code=killed, status=9/KILL
```
__Environment:__
```
# systemctl show influx.service | grep ^Limit
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1073741816
LimitNOFILESoft=1073741816
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=6081
LimitNPROCSoft=6081
LimitMEMLOCK=242425856
LimitMEMLOCKSoft=242425856
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=6081
LimitSIGPENDINGSoft=6081
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
# systemctl show influxdb.service | grep ^Memory
MemoryCurrent=[not set]
MemoryAvailable=infinity
MemoryAccounting=yes
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
MemoryDenyWriteExecute=no
```
__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```
Contributor guide
Research direction
Start by reproducing the Raspberry Pi 4B workload against /api/v2/write with swap disabled, then collect the requested pprof and iostat outputs while TSM compaction is active. Investigate the storage-engine compaction indicated by the logs; done means the same long-running workload no longer causes influxd to be killed by the OOM killer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100