influxdata / influxdata/influxdb
Drop/Delete measurement slow
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Delete/Drop measurement with large number of series cost a lot of time.
Step to reproduce:
Environment: 4C8G
1. Insert 1Million series with influx-stress tool
`influx-stress insert --batch-size 1000 --db stress --series 1000000 --points 268435456`
2. drop measurement ctr
`time influx -database stress -execute "delete from ctr"`
**real 160m22.570s
user 0m0.506s
sys 0m0.034s**
Profile goroutine stack is as followers:
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Influxdb-Build: OSS
X-Influxdb-Version: unknown
Date: Wed, 13 May 2020 07:30:45 GMT
Transfer-Encoding: chunked
1 @ 0x439230 0x407b67 0x40782b 0xa45c58 0xa4795f 0xb01a4c 0xaffb83 0x906cb4
0x469061
# 0xa45c57 github.com/influxdata/influxdb/tsdb.(*Store).walkShards+0xc7
# 0xa4795e github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries+0x4ee
# 0xb01a4b github.com/influxdata/influxdb/coordinator.
(*StatementExecutor).executeDeleteSeriesStatement+0x1db
# 0xaffb82 github.com/influxdata/influxdb/coordinator.
(*StatementExecutor).ExecuteStatement+0x1f52
# 0x906cb3 github.com/influxdata/influxdb/query.(*Executor).executeQuery+0x393`
1 @ 0x4c2665 0x4bfb20 0x4d7882 0x4e284c 0x10f1ed7 0x10f5d7f 0x11122ca 0x1103f89
0x10e9a5c 0x116388c 0x1162b6d 0x116239e 0xa33e6e 0xa52ae9 0xa51e7a 0x469061
# 0x4c2664 syscall.Syscall+0x4
# 0x4bfb1f syscall.Fsync+0x3f
# 0x4d7881 internal/poll.(*FD).Fsync+0x91
# 0x4e284b os.(*File).Sync+0x4b
# 0x10f1ed6 github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).FlushAndSync+0x46
# 0x10f5d7e github.com/influxdata/influxdb/tsdb/index/tsi1.
(*LogFile).DeleteSeriesID+0x13e
# 0x11122c9 github.com/influxdata/influxdb/tsdb/index/tsi1.
(*Partition).DropSeries.func1+0x79
# 0x1103f88 github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).DropSeries+0x38
# 0x10e9a5b github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropSeries+0x9b
# 0x116388b github.com/influxdata/influxdb/tsdb/engine/tsm1.
(*Engine).deleteSeriesRange+0xb8b
# 0x1162b6c github.com/influxdata/influxdb/tsdb/engine/tsm1.
(*Engine).DeleteSeriesRangeWithPredicate+0x79c
# 0x116239d github.com/influxdata/influxdb/tsdb/engine/tsm1.
(*Engine).DeleteSeriesRange+0x7d
# 0xa33e6d github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x8d
# 0xa52ae8 github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x5c8
# 0xa51e79 github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x39`
According to the source code, FlushAndSync() will be called when dropping series one by one, which introduces high iops.
// DeleteSeriesID adds a tombstone for a series id.
func (f *LogFile) DeleteSeriesID(id uint64) error {
f.mu.Lock()
defer f.mu.Unlock()
e := LogEntry{Flag: LogEntrySeriesTombstoneFlag, SeriesID: id}
if err := f.appendEntry(&e); err != nil {
return err
}
f.execEntry(&e)
// Flush buffer and sync to disk.
return f.FlushAndSync()
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tsdb/index/tsi1.(*LogFile).DeleteSeriesID and trace the shown call chain through Partition.DropSeries, Engine.deleteSeriesRange, and Store.DeleteSeries. Reproduce the one-million-series delete with influx-stress and profile the operation. Done means the delete no longer incurs a per-series FlushAndSync cost and its runtime is measurably improved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100