influxdata / influxdata/influxdb

"Failed to open shard" "resource temporarily unavailable" on InfluxDB v1.8.10

Open
#22,937 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We've been trying to migrate from InfluxDB 2 to 1.8, since 2.0 does not have enterprise support. In order to do this, I've been trying to push all our data into an instance of 1.8.
At some point it failed to to the file limit (at that point being 1024) being reached, which is understandable.
I also saw log entries like:
```
ts=2021-11-25T14:02:24.923989Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0Y1r4TXW000 engine=tsm1 service=filestore path=/mnt/InfluxData/1.8/data/logger/autogen/4/000003876-000000001.tsm id=371 error="init: read tombstones: open /mnt/InfluxData/1.8/data/logger/autogen/4/000003876-000000001.tombstone: resource temporarily unavailable"
ts=2021-11-25T14:02:24.944787Z lvl=error msg="Cannot rename corrupt tsm file" log_id=0Y1r4TXW000 engine=tsm1 service=filestore path=/mnt/InfluxData/1.8/data/logger/autogen/4/000003876-000000001.tsm id=371 error="rename /mnt/InfluxData/1.8/data/logger/autogen/4/000003876-000000001.tsm /mnt/InfluxData/1.8/data/logger/autogen/4/000003876-000000001.tsm.bad: resource temporarily unavailable"
```
Suggesting that influx didn't deal well with the file limier having been reached.

After having upped this to 1M files, I started getting the following error, and Influx won't start:
```
ts=2021-11-25T13:51:29.928594Z lvl=info msg="Failed to open shard" log_id=0Y1qYX6G000 service=store trace_id=0Y1qYrfG000 op_name=tsdb_open db_shard_id=6 error="[shard 6] error opening file /mnt/InfluxData/1.8/data/logger/autogen/6/000006247-000000001.tsm: open /mnt/InfluxData/1.8/data/logger/autogen/6/000006247-000000001.tsm: resource temporarily unavailable"
```

And I'm still seeing those "corrupt tsm file" entries in the logs.

__Steps to reproduce:__

Try to push many datapoints to the database while having ulimit -n set too low.
Set ulimit -n higher.
Push more datapoints until influx breaks

__Expected behavior:__
InfluxDB not to fail to serve and be able to start up.

__Actual behavior:__
Influxdb stopped serving and didn't start back up.

__Environment info:__

Linux 5.4.0-89-generic x86_64
InfluxDB v1.8.10 (git: 1.8 688e697c51fd)
ulimit -m = 1M files

__Config:__
[meta]
dir = "/mnt/InfluxData/1.8/meta"
[data]
dir = "/mnt/InfluxData/1.8/data"
wal-dir = "/mnt/InfluxData/1.8/wal"
max-series-per-database = 0
series-id-set-cache-size = 100
[http]
flux-enabled = true
bind-address = ":4046"
max-body-size = 0

__Logs:__
```
ts=2021-11-25T13:51:29.928594Z lvl=info msg="Failed to open shard" log_id=0Y1qYX6G000 service=store trace_id=0Y1qYrfG000 op_name=tsdb_open db_shard_id=6 error="[shard 6] error opening file /mnt/InfluxData/1.8/data/logger/autogen/6/000006247-000000001.tsm: open /mnt/InfluxData/1.8/data/logger/autogen/6/000006247-000000001.tsm: resource temporarily unavailable"
```

__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
Looks like I only get 16 bytes back from influx. Not a valid file:
[profiles.tar.gz](https://github.com/influxdata/influxdb/files/7603682/profiles.tar.gz)

iostat -xd 1 30 > iostat.txt
[iostat.txt](https://github.com/influxdata/influxdb/files/7603684/iostat.txt)
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with a low ulimit -n, then repeat after raising the limit while monitoring the "Failed to open shard" and "corrupt tsm file" log entries. Compare the file-limit behavior with the expected outcome: InfluxDB should continue serving and start successfully after resource exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.