influxdata / influxdata/influxdb

Influx queries fail after Reindexing

Open
#25,254 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We're experiencing InfluxDB outages randomly after `_monitoring` bucket is reindexing shards.

This causes queries to fail and effectively we can't query anything in InfluxDB.

Also, we see shard errors only on the monitoring bucket, but we're unable to query any other bucket.

![image](https://github.com/user-attachments/assets/8423cd8b-4154-414e-9cf9-b09d1e467c6f)

Unfortunately, we have no way to reproduce nor we have any idea why this happens.

Expected behaviour:
No fails / indicative error message

Actual behaviour:

Failing to query all InfluxDB buckets. Worth mentioning we have dbrp configured to support influxql on these buckets.

Config:

InfluxDB version 2.7.5

Default, other than:

```
bolt-path = "/var/lib/influxdb/influxd.bolt"
engine-path = "/influx/engine"
storage-max-concurrent-compactions = 1
```
Log output:

```
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415431Z lvl=info msg="index opened with 8 partitions" log_id=0r6_gaWl000 service=storage-engine index=tsi
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415649Z lvl=info msg="loading changes (start)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415683Z lvl=info msg="loading changes (end)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.036ms
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415927Z lvl=info msg="Reindexing TSM data" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 db_shard_id=612
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415935Z lvl=info msg="Reindexing WAL data" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 db_shard_id=612
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.420356Z lvl=info msg="saving field index changes (start)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.422280Z lvl=info msg="saving field index changes (end)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=1.927ms
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.422313Z lvl=info msg="Write failed" log_id=0r6_gaWl000 service=storage-engine service=write shard=612 error="engine: context canceled"
```

Contributor guide

Open the contributing guide

Research direction

Start with the TSM and WAL reindexing path for shard 612 and the handling of the `engine: context canceled` write failure, using the provided logs and InfluxDB 2.7.5 configuration. Investigate why reindexing `_monitoring` makes queries to other buckets fail; done means queries remain available or return an indicative error, with a regression test if a reproduction is found.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.