TSM compaction causing out of memory error

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. ...TSM compaction starts
2. ...30 seconds or so later "out of memory" error happens
3. ...

__Environment info:__

* System info: Linux 4.15.0-200-generic x86_64
* InfluxDB version: InfluxDB v1.8.10 (git: 1.8 688e697c51fd)
* Ubuntu 18.04 VM On ESXi with 1vCPU and 4GB of RAM

__Logs:__
Dec 14 23:10:12 u1804tig influxd-systemd-start.sh[10648]: fatal error: runtime: out of memory
Dec 14 23:10:12 u1804tig influxd-systemd-start.sh[10648]: runtime stack:
Dec 14 23:10:12 u1804tig influxd-systemd-start.sh[10648]: runtime.throw(0x16bba58, 0x16)
Dec 14 23:10:12 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/panic.go:774 +0x72
Dec 14 23:10:12 u1804tig influxd-systemd-start.sh[10648]: runtime.sysMap(0xc130000000, 0x4000000, 0x3647818)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mheap).sysAlloc(0x362e800, 0xa000, 0x8000, 0x13effed80)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/malloc.go:701 +0x1cd
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mheap).grow(0x362e800, 0x5, 0xffffffffffffffff)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mheap.go:1255 +0xa3
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mheap).allocSpanLocked(0x362e800, 0x5, 0x3647828, 0x7fd1f150b340)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mheap.go:1170 +0x266
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mheap).alloc_m(0x362e800, 0x5, 0x71, 0x7fd1f150b340)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mheap.go:1022 +0xc2
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mheap).alloc.func1()
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mheap.go:1093 +0x4c
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.systemstack(0x0)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.mstart()
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/proc.go:1146
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: goroutine 4539710 [running]:
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.systemstack_switch()
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc061fd2830 sp=0xc061fd2828 pc=0x45ee90
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mheap).alloc(0x362e800, 0x5, 0x10071, 0x7fd1f150b340)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mheap.go:1092 +0x8a fp=0xc061fd2880 sp=0xc061fd2830 pc=0x427b5a
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mcentral).grow(0x36307f8, 0x0)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mcentral.go:255 +0x7b fp=0xc061fd28c0 sp=0xc061fd2880 pc=0x419aab
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mcentral).cacheSpan(0x36307f8, 0x7fd1ddc0d5a0)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mcentral.go:106 +0x2fe fp=0xc061fd2920 sp=0xc061fd28c0 pc=0x4195ce
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mcache).refill(0x7fd25897d008, 0x71)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/mcache.go:138 +0x85 fp=0xc061fd2940 sp=0xc061fd2920 pc=0x419075
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.(*mcache).nextFree(0x7fd25897d008, 0x71, 0x18c0, 0xc12fff9980, 0xc061fd2a38)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/malloc.go:854 +0x87 fp=0xc061fd2978 sp=0xc061fd2940 pc=0x40d9c7
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.mallocgc(0x3500, 0x0, 0x300, 0x318)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/malloc.go:1022 +0x793 fp=0xc061fd2a18 sp=0xc061fd2978 pc=0x40e303
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: runtime.growslice(0x13d2560, 0xc12ffa6f40, 0x317, 0x318, 0x62f, 0x0, 0x0, 0xc12fff9980)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/usr/local/go/src/runtime/slice.go:175 +0x14e fp=0xc061fd2a80 sp=0xc061fd2a18 pc=0x447a7e
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: github.com/influxdata/influxdb/tsdb/cursors.(*IntegerArray).Merge(0xc0321fbb00, 0xc061fd2c40)
Dec 14 23:10:13 u1804tig influxd-systemd-start.sh[10648]: #011/go/src/github.com/influxdata/influxdb/tsdb/cursors/arrayvalues.gen.go:356 +0xa14 fp=0xc061fd2b90 sp=0xc061fd2a80 pc=0x9c4094

....it keeps going...

Every around the same time I receive an out of memory error so I assume it is related to TSM compaction. How can I stop this from happening?

edit

TSM compaction logs:

Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.702632Z lvl=info msg="TSM compaction (start)" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 op_event=start
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703062Z lvl=info msg="Beginning compaction" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_files_n=6
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703208Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000001-000000001.tsm
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703343Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000003-000000001.tsm
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703463Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000004-000000001.tsm
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703588Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000005-000000001.tsm
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703701Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=4 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000006-000000001.tsm
Dec 14 23:09:50 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:50.703814Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=5 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000007-000000001.tsm
Dec 14 23:09:51 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:51.868764Z lvl=info msg="Compacted file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/unifi/autogen/23049/000000007-000000002.tsm.tmp
Dec 14 23:09:51 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:51.869090Z lvl=info msg="Finished compacting files" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 tsm1_files_n=1
Dec 14 23:09:51 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:51.869245Z lvl=info msg="TSM compaction (end)" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RDRW000 op_name=tsm1_compact_group db_shard_id=23049 op_event=end op_elapsed=1166.624ms
Dec 14 23:09:51 u1804tig influxd-systemd-start.sh[10648]: [httpd] 127.0.0.1 - - [14/Dec/2022:23:09:51 -0500] "POST /write?db=sabnzbd HTTP/1.1 " 204 0 "-" "python-requests/2.18.4" 52e65426-7c2e-11ed-ae62-005056ba0622 680
Dec 14 23:09:51 u1804tig influxd-systemd-start.sh[10648]: [httpd] 127.0.0.1 - - [14/Dec/2022:23:09:51 -0500] "POST /write?db=sabnzbd HTTP/1.1 " 204 0 "-" "python-requests/2.18.4" 52e661e1-7c2e-11ed-ae63-005056ba0622 8548
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705473Z lvl=info msg="TSM compaction (start)" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 op_event=start
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705499Z lvl=info msg="Beginning compaction" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_files_n=7
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705502Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000129-000000004.tsm
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705506Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000137-000000002.tsm
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705508Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000145-000000002.tsm
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705511Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000153-000000002.tsm
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705514Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=4 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000154-000000001.tsm
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705517Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=5 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000155-000000001.tsm
Dec 14 23:09:52 u1804tig influxd-systemd-start.sh[10648]: ts=2022-12-15T04:09:52.705520Z lvl=info msg="Compacting file" log_id=0eki37sW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0el~RLGG000 op_name=tsm1_compact_group db_shard_id=23045 tsm1_index=6 tsm1_file=/var/lib/influxdb/data/_internal/monitor/23045/000000156-000000001.tsm

Start with the TSM compaction path and the stack-trace entry at tsdb/cursors/arrayvalues.gen.go, specifically IntegerArray.Merge. Reproduce compaction on the reported InfluxDB v1.8.10 Linux VM with 4GB of RAM, using the supplied logs to correlate the failure. Done means TSM compaction completes without an out-of-memory error.

Written by the indexing model from the issue text.