influxdata / influxdata/influxdb

Influxdb does not start any longer

Open
#21,468 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Maybe this is linked to #21467:

Version 2.06 on Ubuntu 18.04 (virtual environment, 6 CPUs, 16GB RAM)
ulimit -n set to 65535 via /etc/security/limits.conf

Clean install as service, no special config file or parameters

Starting with "systemctl start influxdb"

Had issue "too many open files" before (see #21467)

Service does not start any longer and is stuck in restart loop:

May 13 14:42:14 h2934423 influxd[1633]: ts=2021-05-13T12:42:14.957698Z lvl=info msg="Opened shard" log_id=0U5RlJ50000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/7761a8601e659405/autogen/34650 duration=38.257ms
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.422499Z lvl=info msg="Opened shard" log_id=0U5RlJ50000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/7d4981dee9ebc742/autogen/42194 duration=588.498ms
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.704956Z lvl=info msg="Open store (end)" log_id=0U5RlJ50000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=69525.295ms
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.706039Z lvl=info msg="Starting retention policy enforcement service" log_id=0U5RlJ50000 service=retention check_interval=30m
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.706333Z lvl=info msg="Starting precreation service" log_id=0U5RlJ50000 service=shard-precreation check_interval=10m advance_period=30m
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.707241Z lvl=info msg="Starting query controller" log_id=0U5RlJ50000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.732140Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0U5RlJ50000 max_select_point=0 max_select_series=0 max_select_buckets=0
May 13 14:42:15 h2934423 influxd[1633]: ts=2021-05-13T12:42:15.768969Z lvl=error msg="Failed to start nats streaming server" log_id=0U5RlJ50000 error="nats: no servers available for connection"
May 13 14:42:15 h2934423 influxd[1633]: Error: nats: no servers available for connection
May 13 14:42:15 h2934423 influxd[1633]: See 'influxd -h' for help
May 13 14:42:16 h2934423 systemd[1]: influxdb.service: Main process exited, code=exited, status=1/FAILURE
May 13 14:42:16 h2934423 systemd[1]: influxdb.service: Failed with result 'exit-code'.
May 13 14:42:17 h2934423 systemd[1]: influxdb.service: Service hold-off time over, scheduling restart.
May 13 14:42:17 h2934423 systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 15.

What can I do that the service starts again?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the restart loop with systemctl start influxdb on Ubuntu 18.04 and inspect the service logs, especially the Failed to start nats streaming server error. Done means identifying the cause of the startup failure and confirming that the service starts normally without repeated restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, ubuntu
Domain
backend, databases, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.