influxdata / influxdata/influxdb

[v1] Killed timeout delete blocked new delete

Open
#27,211 6 comments 0 reactions 1 assignee Claimed by @devanbenz View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We have an v1.12.2 influxdb running with index-version = tsi1 on server with 256GB memor/64CPU, and 2/3 of the resource was dedicated to the influxdb.

Due to historic reason, the telegraf db is 36TB (yes, it's TB) with recent shard ~350GB each.
There were ~200 unused measurements incidentally uploaded into this DB, we'd like to drop them.
As expected, dropping measurement crashed the instance, so we ended up having a plan like below:
1. delete all these measurements in a shard.
2. wait the shard to be compacted and continue to other shard (basically delete shard by shard)
3. drop series later when all the TSM data were deleted.

We tried to delete very slowly (only allowing one delete to run at a time) and simple:

delete measurement_name where time>=shard_start_time and time < shard_end_time

At beginning, the whole shard window delete for a measurement ran very fast, returned in less than90s all the time. At the same time, the compactions were kept being monitored, no new deletes will be issued when compactions(full+level 3/2/1) > 3 and active queries (including killed queries) > 12.

However, for some unknown reason, a few deletes timed out, then in a sudden, all new deletes timeout, no matter how tiny or large they are. And this continued to happen even after waited 10s of hours and all the tombstones and *.tmp TSM files for deleted shard were gone.

__Expected behaviour:__
After 10+ hours waiting (no new deletes, and all the existing compactions were cleared), the new deletes should be fast as freshly restarted instance.

__Actual behaviour:__
The problem: after the first few delete timeouts, the later deletes are always timeout, even though all the in-progress compactions(full+level1/2/3) <= 3, active queries < 12 (including the killed deletes) and new delete is tiny. and this happens to delete nothing statement - select with clause returns nothing, but delete with same clause, also timeout.

The "show queries" statement keeps including the killed deletes in its output.

__Guesses:__
The active queries from queryExecutor.queriesActive is more than 8 that is higher than max_concurrent_deletes setting. I suspect that the queryExecutor.queriesActive counted the killed deletes.

__Environment info:__
Linux sdfk8sc050 5.14.0-570.58.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Oct 31 13:55:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux. with wekafs
256GB memory/64CPU.

__Config:__
[data]
enabled = true
dir = "/var/lib/influxdb/data"
index-version = "tsi1"
engine ="tsm1"
wal-dir = "/var/lib/influxdb/wal"
wal-logging-enabled = true
wal-fsync-delay = "1000ms"
query-log-enabled = false
max-wal-size = 104857600
wal-flush-interval = "10m"
wal-partition-flush-delay = "2s"

cache-max-memory-size = 4294967296
cache-snapshot-memory-size = 134217728
compact-throughput = "1024m"
compact-throughput-burst = "4096m"
cache-snapshot-write-cold-duration = "10m"
compact-full-write-cold-duration = "2h"
compact-series-file = true
max-concurrent-compactions = 16
max-concurrent-deletes = 4
max-points-per-block = 0
data-logging-enabled = false
max-series-per-database = 0
max-values-per-tag = 0

[coordinator]
write-timeout = "60s"
max-concurrent-queries = 256
query-timeout = "300s"
log-queries-after = "30s"
max-select-point = 0
max-select-series = 0
max-select-buckets = 0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.