influxdata / influxdata/influxdb
Index out of range in influxdb2 at startup
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Hello,
I'm hitting an interesting error when launching an existent influxdb2 pod in k8s. The stateful set was working great until a recent restart, since then I'm unable to startup the pod properly with the following error:
`ts=2022-04-08T10:09:57.720955Z lvl=info msg="Opened shard" log_id=0_jAFoh0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/8ec72314c1dc0090/autogen/178 duration=9902.121ms
ts=2022-04-08T10:09:57.806059Z lvl=info msg="index opened with 8 partitions" log_id=0_jAFoh0000 service=storage-engine index=tsi
panic: runtime error: index out of range [1] with length 0
goroutine 619 [running]:
encoding/binary.bigEndian.Uint16(...)
/home/circleci/.tools/go/src/encoding/binary/binary.go:101
github.com/influxdata/influxdb/v2/tsdb.ReadSeriesKeyMeasurement(...)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:380
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).execSeriesEntry(0xc00f9ff3b0, 0xc01e68bc68)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:698 +0xb6f
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).execEntry(0xc01e68bc68, 0x7f8c881831c6)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:638 +0x3e
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).open(0xc00f9ff3b0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:166 +0x3cf
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).Open(0xc00f9ff3b0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:115 +0x6b
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).openLogFile(0xc01e79aea0, {0xc0583ca870, 0x404dd20})
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:254 +0x4f
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open(0xc01e79aea0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:197 +0x96f
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Index).Open.func1(0xc01e4b8fc0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:296 +0x47
created by github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Index).Open
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:290 +0x4da
stream closed`
Not sure how to approach this issue, or what can be done to bypass is, so any suggestion would be grateful.
I'm deploying using the latest chart version, in k8s version 1.19.15
Custom values that are provided to the chart are resources(limit cpu to 2, memory to 4Gi), persistence is enabled, ingress in enabled with tls and an admin user is provided to the chart, the rest is default.

Thanks in advance!
Contributor guide
Research direction
Start with the stack-trace entry points in tsdb/series_file.go, especially ReadSeriesKeyMeasurement, and tsdb/index/tsi1/log_file.go around execSeriesEntry and LogFile.open. Reproduce startup with the reported persisted shard and inspect the failing index data. Done means the existing InfluxDB pod opens this index without a panic, with regression coverage for the malformed or empty series key case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100