influxdb consumes all process space and panics, influx database becomes unavailable without loss of data

After running influxdb 1.6.4 under Debian Stable for many years, it crashed last night when it ran out of process address space to allocate more memory. Despite repeated attempts by systemd to restart it, it always failed when something next connected with:

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0xb6d72746 m=0 sigcode=4294967290
goroutine 0 [idle]:
runtime: unknown pc 0xb6d72746
stack: frame={sp:0xbee6f938, fp:0x0} stack=[0xbe670d7c,0xbee6fdc0)
bee6f8b8: 00000000 00000000 0005afbc 00000000
bee6f8c8: 00000000 00000000 00000000 00000000
bee6f8d8: b6e93700 b6d67ecc 0d696914 b6e932d8
bee6f8e8: b6e932d8 b6d5ed98 00000910 7c96f087
bee6f8f8: b6d5edcc b6e81129 00000001 b6d67ecc
bee6f908: 00000910 b6d67fac b6e932d8 bee6f964
bee6f918: bee6f960 00000000 00000000 7c96f087
bee6f928: b6d67fac bee6f960 b6d695f3 03e4b784
bee6f938: <00000006 b6d800af bee6f9d4 00000000
bee6f948: 00000000 aaaaaaab bee6f964 b6e935d8
bee6f958: 00000004 b6ea2970 00000000 00000000
bee6f968: bee6f980 7c96f087 00000000 b6e93700
bee6f978: b6d695f3 bee6fa34 bee6f9d4 bee6f9dc
bee6f988: b6e932d8 b6e81761 bee6f9dc b6ea3ad8
bee6f998: 00000002 b6e93700 00000005 00000000
bee6f9a8: 00000001 b6e932d8 00000000 b6e93494
runtime: unknown pc 0xb6d72746
stack: frame={sp:0xbee6f938, fp:0x0} stack=[0xbe670d7c,0xbee6fdc0)
bee6f8b8: 00000000 00000000 0005afbc 00000000
bee6f8c8: 00000000 00000000 00000000 00000000
bee6f8d8: b6e93700 b6d67ecc 0d696914 b6e932d8
bee6f8e8: b6e932d8 b6d5ed98 00000910 7c96f087
bee6f8f8: b6d5edcc b6e81129 00000001 b6d67ecc
bee6f908: 00000910 b6d67fac b6e932d8 bee6f964
bee6f918: bee6f960 00000000 00000000 7c96f087
bee6f928: b6d67fac bee6f960 b6d695f3 03e4b784
bee6f938: <00000006 b6d800af bee6f9d4 00000000
bee6f948: 00000000 aaaaaaab bee6f964 b6e935d8
bee6f958: 00000004 b6ea2970 00000000 00000000
bee6f968: bee6f980 7c96f087 00000000 b6e93700
bee6f978: b6d695f3 bee6fa34 bee6f9d4 bee6f9dc
bee6f988: b6e932d8 b6e81761 bee6f9dc b6ea3ad8
bee6f998: 00000002 b6e93700 00000005 00000000
bee6f9a8: 00000001 b6e932d8 00000000 b6e93494
goroutine 1 [chan receive]:
main.(*Main).Run(0x197ffac, 0x1884068, 0x2, 0x3, 0xfbcf70, 0x0)
#011github.com/influxdata/influxdb/cmd/influxd/main.go:90 +0x234
main.main()
#011github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x130
goroutine 20 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
...

Searching google came across this post:
https://www.influxdata.com/blog/how-to-overcome-memory-usage-challenges-with-the-time-series-index/

So I rebuilt influxdb 1.6.4 (Debian don't package influx_inspect) and converted the database to tsi1 indicies. This did not improve the situation - now influxdb panics while reading the database during startup:

ts=2021-06-19T10:11:08.445699Z lvl=info msg="Reading file" log_id=0UpwqQUG000 engine=tsm1 service=cacheloader path=/mnt/storage/influxdb/wal/carbon/autogen/604/_00005.wal size=10485990
runtime: out of memory: cannot allocate 8192-byte block (636518400 in use)
fatal error: out of memory
runtime: out of memory: cannot allocate 8192-byte block (636518400 in use)
fatal error: out of memory
runtime stack:
runtime.throw(0x8ca717, 0xd)
#011/usr/lib/go-1.11/src/runtime/panic.go:608 +0x5c
runtime.(*mcache).refill(0xb6b806c0, 0x1432609)
#011/usr/lib/go-1.11/src/runtime/mcache.go:124 +0xe8
runtime.(*mcache).nextFree.func1()
#011/usr/lib/go-1.11/src/runtime/malloc.go:749 +0x24
runtime.systemstack(0x9bf174c0)
#011/usr/lib/go-1.11/src/runtime/asm_arm.s:354 +0x84
runtime.mstart()
#011/usr/lib/go-1.11/src/runtime/proc.go:1229

It is basically consuming all of the processes virtual address space until there are no more regions in the low 3GiB to be given to the process - because it is absolutely full of file maps.

I've tried building influxdb 1.8.6 under Debian Buster, but it fails and I can't work out why. I'm not a go programmer, so I've no idea why it's failing; there seems to be no tool in the go ecosystem to verify that one has the right dependencies available for building.

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Run the database for several years accumulating data
2. Wait

__Expected behavior:__
The processes VM not to constantly expand and expand until there is nothing left. This is why we have storage on disk.

__Actual behavior:__
influxd dies. Database offline from then on and irrecoverable without deleting entire database.

__Environment info:__

* System info: Linux 5.11.0+ armv7l
* InfluxDB version: InfluxDB v1.6.4 (git: unknown unknown)
* Other relevant environment details: Up to date Debian Buster (10)

Start with the influxd entry points in cmd/influxd/main.go and trace startup while it reads the WAL and database files described in the logs. Reproduce on the reported ARMv7 Debian environment with the long-running dataset and inspect virtual-memory and file-map growth. Done means influxd starts and remains available without exhausting address space or panicking during WAL/database loading.

Written by the indexing model from the issue text.