suboptimal behaviour when out of diskspace ("no space left on device")

this was already reported at #10041 #18156 #19650 https://github.com/influxdata/influxdb/issues/9923#issuecomment-533937959 (the oldest report was THREE AND A HALF YEARS AGO!) but the tickets were all closed without investigation of the root-cause.

__Steps to reproduce:__
1. put storage on a too small filesystem
2. log data until filesystem is full
3. observe influxd behaviour
4. make space available again by deleting unrelated files or resizing filesytem
5. observe influxd behavour

__Actual behavior:__
* influxdb rejects writes (http 5xx) BUT still stores the data in in-memory cache.
(clients will retry those writes, leading to increased load and maybe duplicates.)
* influxd will still deliver current data from the cache, masking the disk error.
* system load rises as influxd cache size grows and influxd uses lots of cpu

* behaviour does NOT change AT ALL when space becomes available again
(observing the influxd process with `strace` shows that *NO* errors are returned from
operating system calls, yet influxd keeps logging "no space left on device"!)

* restarting influxd fixes the error, but the data in the in-memory cache is lost.
(which is very surprising to an unsuspecting user - data that seemed stored is suddenly gone.)
(as a workaround, one can dump the affected data (from the point of running
out of space until before the restart) and restore it later.)

__Expected behavior:__
* storing should resume and system load normalize when space is available again

less important:
* unsaved data from memory cache (if any) should be written to disk when space becomes available
* data should either be fully stored, xOR writes rejected, not any other combination of caching, storing, failing.
(different behaviour might be desirable for some users and made available, but should be configurable.)

__Environment info:__

* System info: `Linux 5.10.0-4-amd64 x86_64`
* InfluxDB version: `InfluxDB v1.8.10 (git: 1.8 688e697c51fd)` (from debian sid)

__Config:__
```
# diff /etc/influxdb/influxdb.conf.dpkg-dist /etc/influxdb/influxdb.conf | grep -vE '^.[[:space:]]*(#.*)?$' | grep '^[<>]'
> wal-fsync-delay = "1s"
> log-queries-after = "8s"
> store-enabled = false
> store-database = "_internal"
> store-interval = "60s"
```

__Logs:__
(this log is shortened, but there was no write() returning `ENOSPC` to be seen anywhere...)
```
[pid 6669] read(332, "POST /write?db=vzlogger&precision=ms HTTP/1.1\r\nHost: someserver:8086\r\nAccept: */*\r\nContent-Length: 60\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\nvzlogger_temp,id=000003e6db9e value=21.312000 1644862516494\n", 4096) = 215
[pid 6669] read(332, 0xc00203dc31, 1) = -1 EAGAIN (Resource temporarily unavailable)
[pid 6669] write(2, "[httpd] 192.168.1.135 - - [14/Feb/2022:19:15:16 +0100] \"POST /write?db=vzlogger&precision=ms HTTP/1.1 \" 500 131 \"-\" \"-\" 0f8f76e7-8dc2-11ec-9863-00224da4034b 1543\n", 162) = 162
[pid 6669] write(2, "ts=2022-02-14T18:15:16.589757Z lvl=error msg=\"[500] - \\\"engine: error writing WAL entry: write /var/lib/influxdb/wal/vzlogger/autogen/2928/_00002.wal: no space left on device\\\"\" log_id=0ZeZmFyG000 service=httpd\n", 211) = 211
[pid 6669] write(332, "HTTP/1.1 500 Internal Server Error\r\nContent-Type: application/json\r\nRequest-Id: 0f8f76e7-8dc2-11ec-9863-00224da4034b\r\nX-Influxdb-Build: OSS\r\nX-Influxdb-Error: engine: error writing WAL entry: write /var/lib/influxdb/wal/vzlogger/autogen/2928/_00002.wal: no space left on device\r\nX-Influxdb-Version: 1.8.10\r\nX-Request-Id: 0f8f76e7-8dc2-11ec-9863-00224da4034b\r\nDate: Mon, 14 Feb 2022 18:15:16 GMT\r\nContent-Length: 131\r\n\r\n{\"error\":\"engine: error writing WAL entry: write /var/lib/influxdb/wal/vzlogger/autogen/2928/_00002.wal: no space left on device\"}\n", 550
[pid 6669] <... write resumed>) = 550
[pid 6669] read(332, 0xc0368d5000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 6669] newfstatat(AT_FDCWD, "/var/lib/influxdb/data/grafana/autogen/900/do_not_compact", 0xc006663898, 0) = -1 ENOENT (No such file or directory)
[pid 6669] newfstatat(AT_FDCWD, "/var/lib/influxdb/data/vzlogger/autogen/2928/do_not_compact",
[pid 6669] <... newfstatat resumed>0xc006663968, 0) = -1 ENOENT (No such file or directory)
[pid 6669] write(2, "ts=2022-02-14T18:15:17.674187Z lvl=info msg=\"Cache snapshot (start)\" log_id=0ZeZmFyG000 engine=tsm1 trace_id=0ZfMzLfW000 op_name=tsm1_cache_snapshot op_event=start\n", 164) = 164
[pid 6669] write(2, "ts=2022-02-14T18:15:17.674719Z lvl=info msg=\"Cache snapshot (end)\" log_id=0ZeZmFyG000 engine=tsm1 trace_id=0ZfMzLfW000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=0.555ms\n", 179) = 179
[pid 6669] write(2, "ts=2022-02-14T18:15:17.675081Z lvl=info msg=\"Error writing snapshot\" log_id=0ZeZmFyG000 engine=tsm1 error=\"error opening new segment file for wal (1): write /var/lib/influxdb/wal/grafana/autogen/900/_00001.wal: no space left error opening new segment file for wal (1)on device\"\n", 236) = 236
```

Reproduce the failure using the documented /write endpoint and a filesystem with limited space, then inspect the WAL, cache snapshot, and recovery behavior shown in the logs. Done means writes and system load recover after space is restored, with clear handling of cached data and no restart required.

Written by the indexing model from the issue text.