influxdata / influxdata/influxdb
Panic "index out of range" at startup with "Invalid column reader received from predecessor" after upgrade to 2.1.1
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
After upgrading from 2.0.6 to 2.1.1 our influxdb2 instance (hosted in OpenShift 4.7, deployed with the influxdata/influxdb2:2.0.2 chart) crashes on startup with the following stack trace:
```
ts=2021-12-02T13:48:03.942473Z lvl=info msg="Invalid column reader received from predecessor" log_id=0YArG7rW000 service=storage-reads source="@56:5-57:61: filter" error="mismatched column lengths: map[_field:string:2 _measurement:string:1 _start:time:1 _stop:time:1 _time:time:1 _value:string:1 component:string:1 host:string:1 manufacturer:string:1 model:string:1 pool:string:1 serial:string:1 url:string:1]"
panic: arrow/array: index out of range
goroutine 3179 [running]:
github.com/apache/arrow/go/arrow/array.NewSliceData(0xc0f86304c0, 0xc0feaa5c80, 0xc0d1215540)
/home/circleci/go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20210722123801-4591d76fce28/array/data.go:149 +0x1a5
github.com/apache/arrow/go/arrow/array.NewSlice({0x40d75e8, 0xc0f86304c0}, 0xc0f86304c0, 0xc0f844ba88)
/home/circleci/go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20210722123801-4591d76fce28/array/array.go:165 +0x45
github.com/influxdata/flux/array.Slice({0x40d4518, 0xc0f86304c0}, 0xc0e3d283c0, 0x41d92f)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/array/array.go:186 +0x79
github.com/influxdata/flux/arrow.Slice(...)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/arrow/utils.go:130
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterChunk(0xc0e3d283c0, 0x47c83e, {{{0x40f47c0, 0xc0f860f310}, {0xc0f8465380, 0xd, 0x10}, {0xc0ff578000, 0xd, 0xd}}}, ...)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/stdlib/universe/filter.go:332 +0x8e9
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1({0x40e7bc8, 0xc0f860f4f0})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/stdlib/universe/filter.go:298 +0x1d9
github.com/influxdata/flux/execute.(*consecutiveTransportTable).Do.func1({0x40e7bc8, 0xc0f860f4f0})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/transport.go:537 +0x542
github.com/influxdata/flux/execute.(*ColListTable).Do(0x3eaee60, 0x1)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/table.go:1328 +0x54
github.com/influxdata/flux/execute.(*consecutiveTransportTable).Do(0xc0f8626330, 0xc0ff5645a0)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/transport.go:519 +0x87
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1({0x0, 0xc0e8ea18c0}, 0xc0ff545180)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/stdlib/universe/filter.go:296 +0x147
github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/internal/execute/table/stream.go:109 +0xa6
created by github.com/influxdata/flux/internal/execute/table.StreamWithContext
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/internal/execute/table/stream.go:106 +0x225
```
I suspect that a scheduled task is causing this problem, but even starting the instance with `--no-tasks` has no effect. We don't see any warnings or crashes in 2.0.6.
__Environment info:__
* OpenShift 4.7, influxdbv2 Helm Chart Version 2.0.2
* Almost no customized config
Contributor guide
Research direction
Start with the stack trace in Flux's stdlib/universe/filter.go and arrow/utils.go, then reproduce the startup crash using the reported 2.1.1 upgrade scenario. Investigate the mismatched column lengths and confirm that startup no longer panics when the affected data or query is processed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100