influxdata / influxdata/influxdb

panic: runtime error: slice bounds out of range

Open
#23,953 1 comment 0 reactions 0 assignees View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Due to 100% disk space utilization my influxdb got corrupted and I tried various method to make it live however we are always facing issue of "slice bounds out of range"
__Steps to reproduce:__
docker run -p 8086:8086 -v /db_data/influxdb:/var/lib/influxdb influxdb:1.8

__Expected behavior:__
It should up normally and start serving the data.

__Actual behavior:__
Throwing errors
```
.......
.......
ts=2022-11-25T07:03:22.907711Z lvl=info msg="Finished compacting files" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPP_dW000 op_name=tsm1_compact_group db_shard_id=2876 tsm1_files_n=1
ts=2022-11-25T07:03:22.907718Z lvl=info msg="TSM compaction (end)" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPP_dW000 op_name=tsm1_compact_group db_shard_id=2876 op_event=end op_elapsed=6521.536ms
ts=2022-11-25T07:03:30.062966Z lvl=info msg="Compacted file" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPPZ~l000 op_name=tsm1_compact_group db_shard_id=2888 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/_internal/monitor/2888/000000053-000000002.tsm.tmp
ts=2022-11-25T07:03:30.063000Z lvl=info msg="Finished compacting files" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPPZ~l000 op_name=tsm1_compact_group db_shard_id=2888 tsm1_files_n=1
ts=2022-11-25T07:03:30.063007Z lvl=info msg="TSM compaction (end)" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPPZ~l000 op_name=tsm1_compact_group db_shard_id=2888 op_event=end op_elapsed=13839.343ms
ts=2022-11-25T07:03:30.199978Z lvl=warn msg="Mixed shard index types" log_id=0eNPMdsG000 service=store tsi1_count=15 inmem_count=2 db_instance=_internal
ts=2022-11-25T07:03:31.878323Z lvl=info msg="TSM compaction (end)" log_id=0eNPMdsG000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0eNPPZiW000 op_name=tsm1_compact_group db_shard_id=2542 op_event=end op_elapsed=15727.945ms
panic: runtime error: slice bounds out of range [:23849] with capacity 158

goroutine 13881 [running]:
github.com/influxdata/influxdb/tsdb/cursors.(*IntegerArray).Include(0xc03956cc38, 0x16f7268b6b86d618, 0x16f7280578ec28c8)
/go/src/github.com/influxdata/influxdb/tsdb/cursors/arrayvalues.gen.go:327 +0x25c
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).combineInteger(0xc0e243e000, 0xc07e1ee301, 0xb, 0xc07e1ee3a0, 0x1)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:1339 +0x2f3
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).mergeInteger(0xc0e243e000)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:1282 +0x193
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).merge(0xc0e243e000)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1850 +0x66
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).Next(0xc0e243e000, 0x7f55093dea96)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1716 +0x13eb
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write(0xc03d8d2120, 0xc03c03b540, 0x4b, 0x25aa640, 0xc0e243e000, 0xc0b5573b01, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1141 +0x1dd
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles(0xc03d8d2120, 0x1fa, 0x3, 0xc0826a38c0, 0x4, 0x4, 0x25aa640, 0xc0e243e000, 0x1, 0x25aa640, ...)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1045 +0x18f
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).compact(0xc03d8d2120, 0xc0826a3800, 0xc0826a38c0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:953 +0x40d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).CompactFull(0xc03d8d2120, 0xc0826a38c0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:971 +0x17e
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup(0xc0bc6ca850)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2289 +0x1c72
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply(0xc0bc6ca850)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2266 +0x4d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactHiPriorityLevel.func1(0xc0152ce430, 0xc0301acc80, 0x2, 0xc0bc6ca850)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2183 +0x12e
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactHiPriorityLevel
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2178 +0x123
```

__Environment info:__

* System info: Linux 3.10.0-1160.el7.x86_64 x86_64

__Config:__
Default one

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Docker launch with the affected /var/lib/influxdb data, then read tsdb/cursors/arrayvalues.gen.go at IntegerArray.Include and tsdb/engine/tsm1/compact.gen.go at combineInteger. Trace the compaction path shown in compact.go and engine.go; done means the corrupted store no longer panics during startup and can serve its data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.