influxdata / influxdata/influxdb

Panic "index out of range" at startup with "Invalid column reader received from predecessor" after upgrade to 2.1.1

Open
#22,960 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

After upgrading from 2.0.6 to 2.1.1 our influxdb2 instance (hosted in OpenShift 4.7, deployed with the influxdata/influxdb2:2.0.2 chart) crashes on startup with the following stack trace:

```
ts=2021-12-02T13:48:03.942473Z lvl=info msg="Invalid column reader received from predecessor" log_id=0YArG7rW000 service=storage-reads source="@56:5-57:61: filter" error="mismatched column lengths: map[_field:string:2 _measurement:string:1 _start:time:1 _stop:time:1 _time:time:1 _value:string:1 component:string:1 host:string:1 manufacturer:string:1 model:string:1 pool:string:1 serial:string:1 url:string:1]"
panic: arrow/array: index out of range

goroutine 3179 [running]:
github.com/apache/arrow/go/arrow/array.NewSliceData(0xc0f86304c0, 0xc0feaa5c80, 0xc0d1215540)
/home/circleci/go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20210722123801-4591d76fce28/array/data.go:149 +0x1a5
github.com/apache/arrow/go/arrow/array.NewSlice({0x40d75e8, 0xc0f86304c0}, 0xc0f86304c0, 0xc0f844ba88)
/home/circleci/go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20210722123801-4591d76fce28/array/array.go:165 +0x45
github.com/influxdata/flux/array.Slice({0x40d4518, 0xc0f86304c0}, 0xc0e3d283c0, 0x41d92f)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/array/array.go:186 +0x79
github.com/influxdata/flux/arrow.Slice(...)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/arrow/utils.go:130
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterChunk(0xc0e3d283c0, 0x47c83e, {{{0x40f47c0, 0xc0f860f310}, {0xc0f8465380, 0xd, 0x10}, {0xc0ff578000, 0xd, 0xd}}}, ...)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/stdlib/universe/filter.go:332 +0x8e9
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1({0x40e7bc8, 0xc0f860f4f0})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/stdlib/universe/filter.go:298 +0x1d9
github.com/influxdata/flux/execute.(*consecutiveTransportTable).Do.func1({0x40e7bc8, 0xc0f860f4f0})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/transport.go:537 +0x542
github.com/influxdata/flux/execute.(*ColListTable).Do(0x3eaee60, 0x1)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/table.go:1328 +0x54
github.com/influxdata/flux/execute.(*consecutiveTransportTable).Do(0xc0f8626330, 0xc0ff5645a0)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/transport.go:519 +0x87
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1({0x0, 0xc0e8ea18c0}, 0xc0ff545180)
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/stdlib/universe/filter.go:296 +0x147
github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/internal/execute/table/stream.go:109 +0xa6
created by github.com/influxdata/flux/internal/execute/table.StreamWithContext
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/internal/execute/table/stream.go:106 +0x225
```

I suspect that a scheduled task is causing this problem, but even starting the instance with `--no-tasks` has no effect. We don't see any warnings or crashes in 2.0.6.

__Environment info:__

* OpenShift 4.7, influxdbv2 Helm Chart Version 2.0.2
* Almost no customized config

Contributor guide

Open the contributing guide

Research direction

Start with the stack trace in Flux's stdlib/universe/filter.go and arrow/utils.go, then reproduce the startup crash using the reported 2.1.1 upgrade scenario. Investigate the mismatched column lengths and confirm that startup no longer panics when the affected data or query is processed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.