influxdata / influxdata/influxdb

InfluxDB API at http://localhost:8086/ready unavailable perpetually

Open
#24,332 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

area/2.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We built an influx v2.7 on Ubuntu with default settings. The only thing we did was point the engine to a new drive.
This database worked fine for weeks and we even loaded it with 12 TB of data. However, it was left alone for a few weeks and when the users came back, the DB was unresponsive.

Now, no matter what we do, we cannot get the service to start successfully and the logs just show that InfluxDB API at http://localhost:8086/ready unavailable ....

We tried adjusting the config file and beefing it up past the defaults.
We tried commenting out Type=forking in the systemd service file
We confirmed that the settings seem to match out other working instances
Restarting the machine and the service
We tried a lot of odds an and ends and cant seem to get it to start

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. deploy influxdb2 on Ubuntu with default config
2. use it, load 12 TB data, leave it
3. upon unresponsiveness, try to restart it to get the service active

__Expected behavior:__
Influx to work normally

__Actual behavior:__
Influx service and database will not start or become responsive

__Environment info:__

* System info: Linux 5.15.0-78-generic x86_64
* InfluxDB version: InfluxDB v2.7.1 (git: 407fa622e9) build_date: 2023-04-28T13:24:27Z
* Other relevant environment details: ubuntu VM on vsphere; bolt on standard drive (100 GB) and engine on separate drive (15 TB); 64 vCPU; 256 GB RAM.

__Config:__
Original config we had when the break happened:
`cat /etc/influxdb/config.toml`
```
bolt-path = "/var/lib/influxdb/influxd.bolt"
engine-path = "/mnt/data/influxdb/engine"
```

Updated config for more resources, still not working:
`cat /etc/influxdb/config.toml`
```
bolt-path = "/var/lib/influxdb/influxd.bolt"
engine-path = "/mnt/data/influxdb/engine"
http-bind-address = ":8086"
tracing-type = "log"
storage-cache-max-memory-size = 107374182400
storage-max-concurrent-compactions = 32
storage-compact-throughput-burst = 1073741824
query-concurrency = 64
query-queue-size = 100
query-initial-memory-bytes = 2147483648
query-max-memory-bytes = 137438953472
session-length = 120
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the systemd service status and logs for the InfluxDB v2.7.1 instance, then compare the reported config paths and storage layout with a working instance. The issue does not name a source file or test and provides no startup log details, so the failure and a verifiable fix still need to be narrowed down.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, ubuntu
Domain
databases, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.