influxdata / influxdata/influxdb
Influx queries fail after Reindexing
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
We're experiencing InfluxDB outages randomly after `_monitoring` bucket is reindexing shards.
This causes queries to fail and effectively we can't query anything in InfluxDB.
Also, we see shard errors only on the monitoring bucket, but we're unable to query any other bucket.

Unfortunately, we have no way to reproduce nor we have any idea why this happens.
Expected behaviour:
No fails / indicative error message
Actual behaviour:
Failing to query all InfluxDB buckets. Worth mentioning we have dbrp configured to support influxql on these buckets.
Config:
InfluxDB version 2.7.5
Default, other than:
```
bolt-path = "/var/lib/influxdb/influxd.bolt"
engine-path = "/influx/engine"
storage-max-concurrent-compactions = 1
```
Log output:
```
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415431Z lvl=info msg="index opened with 8 partitions" log_id=0r6_gaWl000 service=storage-engine index=tsi
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415649Z lvl=info msg="loading changes (start)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415683Z lvl=info msg="loading changes (end)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.036ms
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415927Z lvl=info msg="Reindexing TSM data" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 db_shard_id=612
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.415935Z lvl=info msg="Reindexing WAL data" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 db_shard_id=612
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.420356Z lvl=info msg="saving field index changes (start)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.422280Z lvl=info msg="saving field index changes (end)" log_id=0r6_gaWl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=1.927ms
Aug 20 06:59:14 influxd-systemd-start.sh[1306980]: ts=2024-08-20T06:59:14.422313Z lvl=info msg="Write failed" log_id=0r6_gaWl000 service=storage-engine service=write shard=612 error="engine: context canceled"
```
Contributor guide
Research direction
Start with the TSM and WAL reindexing path for shard 612 and the handling of the `engine: context canceled` write failure, using the provided logs and InfluxDB 2.7.5 configuration. Investigate why reindexing `_monitoring` makes queries to other buckets fail; done means queries remain available or return an indicative error, with a regression test if a reproduction is found.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100