influxdata / influxdata/influxdb

2.0.7 Compacting file result in OOM on Raspberry Pi 4B

Open
#22,124 0 comments 0 reactions 0 assignees View on GitHub
area/2.x area/memory area/performance area/storage
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__

1. Disable swap
2. Insert new measurement every second using `socat` to `/api/v2/write` via python script
3. Wait 23 days

__Expected behavior:__
No OOM

__Actual behavior:__
influxd OOM

__Environment info:__

* Linux 5.10.48-1-ARCH aarch64
* InfluxDB version: InfluxDB 2.0.7 (git: 2a45f0c037) build_date: 2021-06-04T19:17:40Z
* Disk usage:
```Filesystem Size Used Avail Use% Mounted on
dev 665M 0 665M 0% /dev
run 925M 744K 925M 1% /run
/dev/mmcblk0p2 117G 11G 101G 10% /
tmpfs 925M 0 925M 0% /dev/shm
tmpfs 925M 0 925M 0% /tmp
/dev/mmcblk0p1 240M 43M 198M 18% /boot
tmpfs 185M 0 185M 0% /run/user/0
```

__Logs:__
```
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357097Z lvl=info msg="TSM compaction (start)" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy>
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357871Z lvl=info msg="Beginning compaction" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=l>
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357929Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357967Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358003Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358039Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358074Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358113Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358148Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358182Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:27 influxPi4B kernel: influxd invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Jul 31 11:01:27 influxPi4B kernel: CPU: 3 PID: 17748 Comm: influxd Tainted: G C 5.10.48-1-ARCH #1
Jul 31 11:01:27 influxPi4B kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
Jul 31 11:01:27 influxPi4B kernel: Call trace:
Jul 31 11:01:27 influxPi4B kernel: dump_backtrace+0x0/0x1a0
Jul 31 11:01:27 influxPi4B kernel: show_stack+0x18/0x24
Jul 31 11:01:27 influxPi4B kernel: dump_stack+0xc8/0x120
Jul 31 11:01:27 influxPi4B kernel: dump_header+0x48/0x1f4
Jul 31 11:01:27 influxPi4B kernel: oom_kill_process+0x204/0x20c
Jul 31 11:01:27 influxPi4B kernel: out_of_memory+0xe4/0x320
Jul 31 11:01:27 influxPi4B kernel: __alloc_pages_slowpath.constprop.0+0x850/0xba0
Jul 31 11:01:27 influxPi4B kernel: __alloc_pages_nodemask+0x298/0x2f0
Jul 31 11:01:27 influxPi4B kernel: handle_mm_fault+0x4dc/0xc90
Jul 31 11:01:27 influxPi4B kernel: do_page_fault+0x140/0x3d4
Jul 31 11:01:27 influxPi4B kernel: do_translation_fault+0xb0/0xcc
Jul 31 11:01:27 influxPi4B kernel: do_mem_abort+0x44/0xa4
Jul 31 11:01:27 influxPi4B kernel: el0_da+0x28/0x34
Jul 31 11:01:27 influxPi4B kernel: el0_sync_handler+0x168/0x1b0
Jul 31 11:01:27 influxPi4B kernel: el0_sync+0x180/0x1c0
Jul 31 11:01:27 influxPi4B kernel: Mem-Info:
Jul 31 11:01:27 influxPi4B kernel: active_anon:206675 inactive_anon:243243 isolated_anon:0
active_file:239 inactive_file:308 isolated_file:32
unevictable:0 dirty:0 writeback:0
slab_reclaimable:4988 slab_unreclaimable:5648
mapped:442 shmem:179 pagetables:1405 bounce:0
free:3532 free_pcp:31 free_cma:215
Jul 31 11:01:27 influxPi4B kernel: Node 0 active_anon:826700kB inactive_anon:972972kB active_file:956kB inactive_file:1232kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1768>
Jul 31 11:01:27 influxPi4B kernel: DMA free:8144kB min:6648kB low:7524kB high:8400kB reserved_highatomic:0KB active_anon:529016kB inactive_anon:332500kB active_file:200kB inactive_file:204kB u>
Jul 31 11:01:27 influxPi4B kernel: lowmem_reserve[]: 0 984 984 984
Jul 31 11:01:27 influxPi4B kernel: DMA32 free:5984kB min:10340kB low:11344kB high:12348kB reserved_highatomic:0KB active_anon:297424kB inactive_anon:640448kB active_file:876kB inactive_file:14>
Jul 31 11:01:27 influxPi4B kernel: lowmem_reserve[]: 0 0 0 0
Jul 31 11:01:27 influxPi4B kernel: DMA: 358*4kB (UMEC) 241*8kB (UMEC) 129*16kB (UMEC) 59*32kB (UEC) 16*64kB (UEC) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8336kB
Jul 31 11:01:27 influxPi4B kernel: DMA32: 364*4kB (UM) 161*8kB (UME) 174*16kB (UE) 24*32kB (UME) 2*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6424kB
Jul 31 11:01:27 influxPi4B kernel: 725 total pagecache pages
Jul 31 11:01:27 influxPi4B kernel: 0 pages in swap cache
Jul 31 11:01:27 influxPi4B kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 31 11:01:27 influxPi4B kernel: Free swap = 0kB
Jul 31 11:01:27 influxPi4B kernel: Total swap = 0kB
Jul 31 11:01:27 influxPi4B kernel: 504832 pages RAM
Jul 31 11:01:27 influxPi4B kernel: 0 pages HighMem/MovableOnly
Jul 31 11:01:27 influxPi4B kernel: 31338 pages reserved
Jul 31 11:01:27 influxPi4B kernel: 65536 pages cma reserved
Jul 31 11:01:27 influxPi4B kernel: Tasks state (memory values in pages):
Jul 31 11:01:27 influxPi4B kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jul 31 11:01:27 influxPi4B kernel: [ 225] 0 225 3952 190 61440 0 0 systemd-userdbd
Jul 31 11:01:27 influxPi4B kernel: [ 229] 0 229 4366 403 73728 0 -1000 systemd-udevd
Jul 31 11:01:27 influxPi4B kernel: [ 233] 981 233 5397 264 73728 0 0 systemd-network
Jul 31 11:01:27 influxPi4B kernel: [ 311] 980 311 4973 481 69632 0 0 systemd-resolve
Jul 31 11:01:27 influxPi4B kernel: [ 312] 979 312 22392 205 65536 0 0 systemd-timesyn
Jul 31 11:01:27 influxPi4B kernel: [ 315] 81 315 3554 337 57344 0 -900 dbus-daemon
Jul 31 11:01:27 influxPi4B kernel: [ 316] 0 316 1880 134 49152 0 0 socat
Jul 31 11:01:27 influxPi4B kernel: [ 318] 0 318 4067 225 57344 0 0 systemd-homed
Jul 31 11:01:27 influxPi4B kernel: [ 320] 0 320 4088 245 65536 0 0 systemd-logind
Jul 31 11:01:27 influxPi4B kernel: [ 322] 0 322 99907 5174 131072 0 0 python
Jul 31 11:01:27 influxPi4B kernel: [ 323] 0 323 2403 198 49152 0 -1000 sshd
Jul 31 11:01:27 influxPi4B kernel: [ 325] 0 325 2116 129 45056 0 0 agetty
Jul 31 11:01:27 influxPi4B kernel: [ 17654] 976 17654 2059639 437575 4632576 0 0 influxd
Jul 31 11:01:27 influxPi4B kernel: [ 17658] 0 17658 8115 227 69632 0 -250 systemd-journal
Jul 31 11:01:27 influxPi4B kernel: [ 38555] 0 38555 4079 188 61440 0 0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: [ 38556] 0 38556 4079 187 65536 0 0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: [ 38557] 0 38557 4079 188 57344 0 0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=influxd,pid=17654,uid=976
Jul 31 11:01:27 influxPi4B kernel: Out of memory: Killed process 17654 (influxd) total-vm:8238556kB, anon-rss:1749696kB, file-rss:604kB, shmem-rss:0kB, UID:976 pgtables:4524kB oom_score_adj:0
Jul 31 11:01:27 influxPi4B kernel: oom_reaper: reaped process 17654 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 31 11:01:27 influxPi4B systemd[1]: influxdb.service: Main process exited, code=killed, status=9/KILL
```

__Environment:__
```
# systemctl show influx.service | grep ^Limit
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1073741816
LimitNOFILESoft=1073741816
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=6081
LimitNPROCSoft=6081
LimitMEMLOCK=242425856
LimitMEMLOCKSoft=242425856
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=6081
LimitSIGPENDINGSoft=6081
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
# systemctl show influxdb.service | grep ^Memory
MemoryCurrent=[not set]
MemoryAvailable=infinity
MemoryAccounting=yes
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
MemoryDenyWriteExecute=no
```

__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Raspberry Pi 4B workload against /api/v2/write with swap disabled, then collect the requested pprof and iostat outputs while TSM compaction is active. Investigate the storage-engine compaction indicated by the logs; done means the same long-running workload no longer causes influxd to be killed by the OOM killer.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.