Influxdb stops writing

I am running several large influxdb servers, each has about 3-5TB of data, and around 24 CPUs / 156GB RAM. These are running 1.7.10 on Docker.

Recently we have seen major problems with three of these servers, where the servers simply stop accepting writes. The error message sent back was that the write limit had been hit. After restarting the server, writes got in for a short period of time - sometimes up to a few minutes - and then the same thing happened. We could see Write I/O drop on the disk, and errors going back to the writers.

We tried dropping some old shards, and doing verification of the database files. There were no logs that indicated any internal problem and nothing we tried solved this - in each case, once it got into this state, we couldn't find a way out of it.

We ended up having to re-initialize the databases that hit this condition, and manually migrate data back in from a separate recovery instance of influxdb with the old disk mounted; reads worked fine, but not writes.

Given this description, do you have any ideas of what could have happened? Is there anything I can do to investigate further? It will be difficult to reproduce the writing to these disks now but I do still have the disks since they are being used for data recovery.

No source files or tests are identified in the report. Start by examining the retained disks and the InfluxDB 1.7.10 Docker deployment around the write-limit error, failed writes, and available logs; done means identifying a supported cause or a reproducible investigation path without reinitializing the database.

Written by the indexing model from the issue text.