influxdata / influxdata/influxdb

2.0.6 suddenly stopped responding because of too many open files

Open
#24,057 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I have influxdb 2.0.6 running on a raspberry pi quite happily for nearly 2 years, but this morning I noticed it had stopped responding. Even after a reboot it is not happy. I can't get any connection via the web interface (it justs shows the swirling circle) nor via the HTTP API, nor via influx CLI (e.g. influx bucket list just hangs there).

Can anyone advise any more steps to try? Influxd seems to startup fine without error.

__Environment info:__

Linux 5.10.17-v8+ aarch64
InfluxDB 2.0.6 (git: 4db98b4c9a) build_date: 2021-04-29T16:48:12Z
Database files are all on an SSD drive

__Config:__
/usr/local/bin/influxd --bolt-path=/ssd/influx/influxd.bolt --engine-path=/ssd/influx/engine --reporting-disabled --storage-retention-check-interval=24h --log-level=debug

__Logs:__
```
Jan 23 09:36:01 hass systemd[1]: Started influxdb.
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.784207Z lvl=info msg="Welcome to InfluxDB" log_id=0fZVcsml000 version=2.0.6 commit=4db98b4c9a build_date=2021-04-29T16:48:12Z
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.788758Z lvl=info msg="Resources opened" log_id=0fZVcsml000 service=bolt path=/ssd/influx/influxd.bolt
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811082Z lvl=debug msg="buckets find" log_id=0fZVcsml000 store=new took=0.371ms
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811169Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0fZVcsml000 bolt_path=/ssd/influx/influxd.bolt
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811754Z lvl=info msg="Using data dir" log_id=0fZVcsml000 service=storage-engine service=store path=/ssd/influx/engine/data
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811825Z lvl=info msg="Compaction settings" log_id=0fZVcsml000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throug
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.811855Z lvl=info msg="Open store (start)" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open op_event=start
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.883222Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.930022Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.932094Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.945803Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.956105Z lvl=info msg="Opened file" log_id=0fZVcsml000 service=storage-engine engine=tsm1 service=filestore path=/ssd/influx/engine/data/540a390190697254/autogen/1047/00000
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.963548Z lvl=info msg="Opened file" log_id=0fZVcsml000 service=storage-engine engine=tsm1 service=filestore path=/ssd/influx/engine/data/540a390190697254/autogen/111/000000
Jan 23 09:36:35 hass influxd[1204]: ts=2023-01-23T09:36:35.983334Z lvl=info msg="Opened file" log_id=0fZVcsml000 service=storage-engine engine=tsm1 service=filestore path=/ssd/influx/engine/data/540a390190697254/autogen/1081/00000
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055497Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/514933d6df66d
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055515Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/540a390190697
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055532Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/540a390190697
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.055500Z lvl=info msg="Opened shard" log_id=0fZVcsml000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/ssd/influx/engine/data/540a390190697
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.091027Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
Jan 23 09:36:36 hass influxd[1204]: ts=2023-01-23T09:36:36.102203Z lvl=info msg="index opened with 8 partitions" log_id=0fZVcsml000 service=storage-engine index=tsi
```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied systemd startup command and debug logs, then reproduce the hanging web interface, HTTP API, and influx CLI calls while investigating the reported open-file condition. Done means identifying the cause of the unresponsive InfluxDB 2.0.6 service and documenting or fixing the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, raspberry-pi
Domain
databases, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.