influxdata / influxdata/influxdb
[2.x] Index file compaction and delete series range may deadlock each other
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
This was observed in 2.x, but may also apply to 1.x.
## Description
An index file compaction and a delete series range can deadlock each other.
The compaction requires closing the original (input) log file. This waits for all readers on the log file to release their reference counts.
https://github.com/influxdata/influxdb/blob/b88e74e6bbcc72ed765e71675e561ad081d466a7/tsdb/index/tsi1/partition.go#L1358-L1361
Meanwhile `DeleteSeriesRangeWithPredicate` makes a series iterator which has retained a reference to the index file, and then requests that all index compactions are disabled _and waits for compactions to complete_.
https://github.com/influxdata/influxdb/blob/efebf4d569ec2d355db7f05958a09dde080e6a16/tsdb/engine/tsm1/engine.go#L1399-L1404
The delete code path is only one that waits on index log file compaction to stop with `tsiIndex.Wait()`
The order of these events may cause a deadlock.
- Compaction: start log file compaction
- Delete: start delete and retain log file
- Delete: disable compactions and wait for compactions to complete
- Compaction: finish compaction of new file and want for readers to release to remove old file
- Deadlock: Compaction is waiting for the delete process to release the index file and the delete is waiting for the compaction to decrement the in progress compactions
I believe this will _always_ deadlock if the delete creates the iterators and an index compaction is already in progress.
## possible remediations
- a time out for the compaction which might leave orphaned index files or interrupting the compaction when a delete arrives, like what is done for tsm compactions.
- deletes need to check much earlier if an index compaction is running and disable them before retaining any references to index files.
- deletes need a timeout waiting for compactions to complete and return an error to the client to try your delete again soon
The last seems preferable in the short term.
### references
* loosely related to https://github.com/influxdata/influxdb/issues/11586
Contributor guide
Research direction
Start by tracing the compaction path in tsdb/index/tsi1/partition.go around lines 1358-1361 and DeleteSeriesRangeWithPredicate in tsdb/engine/tsm1/engine.go around lines 1399-1404. Confirm the reference-count and tsiIndex.Wait ordering, then establish which proposed remediation prevents the deadlock while preserving safe index-file cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100