Newer Shards Grow To Thousands of TSM Files, Server Holds 65k Files Open (linux erroring Too Many Open Files)

**System Details:**
Centos 7.5

**The version of InfluxDB you are running**
1.8.6

**Whether you installed it using a pre-built package, or built it from source.**
Pre-Built from Yum

**Clear steps to reproduce the issue described, if at all possible.**
We are running a single-node influx server with a large amount of data (~14 TB). Recently, our shards started to blow up in terms of the number of tsm files per shard. In previous shards, there would typically be about 100 to 500 tsm files. In the most recent two shards, there are 43k and 1600 files per shard. This is causing the server to use up the max num open files configured in /etc/security/limits.conf. I have set that to 65k and we are still hitting the limit. This renders the server completely non-functional.

The other side effect of this issue is that queries in the time range of the affected shards seem to be significantly slower and use much more memory. Also, restarting the server now takes nearly 30 minutes, the two affected shards are very slow to open:

> Oct 13 13:30:32 awsuse1-posttrade2.int.lonk.com influxd[74416]: ts=2021-10-13T13:30:32.000103Z lvl=info msg="Opened shard" log_id=0XASy3y0000 service=store trace_id=0XASyK_0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/tradedata/autogen/764 duration=207644.226ms
Oct 13 13:32:04 awsuse1-posttrade2.int.lonk.com influxd[74416]: ts=2021-10-13T13:32:04.783594Z lvl=info msg="Opened shard" log_id=0XASy3y0000 service=store trace_id=0XASyK_0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/tradedata/autogen/748 duration=484899.393ms

We have had some changes in the last few weeks related to our write patterns:
1. Added two new measurements with about 3k cardinality and 50k cardinality respectively
2. The two new measurements are written in a streaming manner (about 100k points per minute), whereas previously we would only write data once per day in bulk (we still do that in addition to the streaming data)

We do all writing via the rest API.

This is an urgent issue for us because the database can barely run.

Start by reviewing the reported shard paths, TSM file counts, open-file limit, and shard-open durations on CentOS 7.5. Reproduce or isolate the behavior with the two streaming measurements and confirm completion by identifying why file usage and shard-opening time grow beyond operational limits.

Written by the indexing model from the issue text.