influxdata / influxdata/influxdb

Index out of range in influxdb2 at startup

Open
#23,266 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hello,
I'm hitting an interesting error when launching an existent influxdb2 pod in k8s. The stateful set was working great until a recent restart, since then I'm unable to startup the pod properly with the following error:
`ts=2022-04-08T10:09:57.720955Z lvl=info msg="Opened shard" log_id=0_jAFoh0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/8ec72314c1dc0090/autogen/178 duration=9902.121ms
ts=2022-04-08T10:09:57.806059Z lvl=info msg="index opened with 8 partitions" log_id=0_jAFoh0000 service=storage-engine index=tsi
panic: runtime error: index out of range [1] with length 0
goroutine 619 [running]:
encoding/binary.bigEndian.Uint16(...)
/home/circleci/.tools/go/src/encoding/binary/binary.go:101
github.com/influxdata/influxdb/v2/tsdb.ReadSeriesKeyMeasurement(...)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:380
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).execSeriesEntry(0xc00f9ff3b0, 0xc01e68bc68)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:698 +0xb6f
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).execEntry(0xc01e68bc68, 0x7f8c881831c6)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:638 +0x3e
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).open(0xc00f9ff3b0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:166 +0x3cf
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).Open(0xc00f9ff3b0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:115 +0x6b
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).openLogFile(0xc01e79aea0, {0xc0583ca870, 0x404dd20})
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:254 +0x4f
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open(0xc01e79aea0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:197 +0x96f
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Index).Open.func1(0xc01e4b8fc0)
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:296 +0x47
created by github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Index).Open
/home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:290 +0x4da
stream closed`

Not sure how to approach this issue, or what can be done to bypass is, so any suggestion would be grateful.

I'm deploying using the latest chart version, in k8s version 1.19.15
Custom values that are provided to the chart are resources(limit cpu to 2, memory to 4Gi), persistence is enabled, ingress in enabled with tls and an admin user is provided to the chart, the rest is default.
Screenshot 2022-04-08 at 11 59 02

Thanks in advance!

Contributor guide

Open the contributing guide

Research direction

Start with the stack-trace entry points in tsdb/series_file.go, especially ReadSeriesKeyMeasurement, and tsdb/index/tsi1/log_file.go around execSeriesEntry and LogFile.open. Reproduce startup with the reported persisted shard and inspect the failing index data. Done means the existing InfluxDB pod opens this index without a panic, with regression coverage for the malformed or empty series key case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.