influxdata / influxdata/influxdb
2.0.6 suddenly stopped responding because of too many open files
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
I have influxdb 2.0.6 running on a raspberry pi quite happily for nearly 2 years, but this morning I noticed it had stopped responding. Even after a reboot it is not happy. I can't get any connection via the web interface (it justs shows the swirling circle) nor via the HTTP API, nor via influx CLI (e.g. influx bucket list just hangs there).
Can anyone advise any more steps to try? Influxd seems to startup fine without error.
__Environment info:__
Linux 5.10.17-v8+ aarch64
InfluxDB 2.0.6 (git: 4db98b4c9a) build_date: 2021-04-29T16:48:12Z
Database files are all on an SSD drive
__Config:__
/usr/local/bin/influxd --bolt-path=/ssd/influx/influxd.bolt --engine-path=/ssd/influx/engine --reporting-disabled --storage-retention-check-interval=24h --log-level=debug
__Logs:__
```
Jan 23 09:36:01 hass systemd[1]: Started influxdb.
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.784207Z lvl=info msg="Welcome to InfluxDB" log_id=0fZVcsml000 version=2.0.6 commit=4db98b4c9a build_date=2021-04-29T16:48:12Z
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.788758Z lvl=info msg="Resources opened" log_id=0fZVcsml000 service=bolt path=/ssd/influx/influxd.bolt
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811082Z lvl=debug msg="buckets find" log_id=0fZVcsml000 store=new took=0.371ms
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811169Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0fZVcsml000 bolt_path=/ssd/influx/influxd.bolt
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811754Z lvl=info msg="Using data dir" log_id=0fZVcsml000 service=storage-engine service=store path=/ssd/influx/engine/data
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811825Z lvl=info msg="Compaction settings" log_id=0fZVcsml000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throug
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811855Z lvl=info msg="Open store (start)" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open op_event=start
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.883222Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.930022Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.932094Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.945803Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.956105Z lvl=info msg="Opened file" log_id=0fZVcsml000 service=storage-engine engine=tsm1 service=filestore path=/ssd/influx/engine/data/540a390190697254/autogen/1047/00000
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.963548Z lvl=info msg="Opened file" log_id=0fZVcsml000 service=storage-engine engine=tsm1 service=filestore path=/ssd/influx/engine/data/540a390190697254/autogen/111/000000
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.983334Z lvl=info msg="Opened file" log_id=0fZVcsml000 service=storage-engine engine=tsm1 service=filestore path=/ssd/influx/engine/data/540a390190697254/autogen/1081/00000
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055497Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/514933d6df66d
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055515Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/540a390190697
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055532Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/540a390190697
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055500Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/540a390190697
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.091027Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.102203Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
```
Contributor guide
Research direction
Start with the supplied systemd startup command and debug logs, then reproduce the hanging web interface, HTTP API, and influx CLI calls while investigating the reported open-file condition. Done means identifying the cause of the unresponsive InfluxDB 2.0.6 service and documenting or fixing the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, raspberry-pi
- Domain
- databases, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100