influxdata / influxdata/influxdb
Bucket delete, create, write causes occasional panic
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
I am using influxdb 2.1.1 in a kubernetes pod. We have a script that deletes a bucket, creates it, and writes configuration values to it (synchronously). Occasionally, this script was getting `('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))` error codes. Adding retry logic, we started seeing `111: Err connection refused` errors. This connection refused was due to the pod panicing, and this panic behavior was reproduced manually as well.
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Start influxdb pod
2. Write values to bucket from tar.gz point-protocol file
3. Drop bucket
4. Create bucket
5. Repeat steps 2-4 until panic happens
__Expected behavior:__
No panic should occur. Bucket create/delete and writing from file should report if it failed, but not cause pod panic.
__Actual behavior:__
The pod will panic occasionally. This happens during test teardown once or twice out of 12 times. While doing this process manually, it took about 8 times to trigger.
__Environment info:__
* Linux 4.15.0-163-generic x86_64
* InfluxDB v2.1.1 (git: 657e1839) build_date: 2022-05-10T14:36:50Z
* Kubernetes pod
__Config:__
```
influx config list --json
{}
```
__Logs:__
```
ts=2022-05-09T22:10:56.342751Z lvl=info msg="index opened with 8 partitions" log_id=0aMfkUnG000 service=storage-engine index=tsi
ts=2022-05-09T22:10:56.343895Z lvl=info msg="Reindexing TSM data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=28
ts=2022-05-09T22:10:56.343915Z lvl=info msg="Reindexing WAL data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=28
ts=2022-05-09T22:11:49.325626Z lvl=info msg="index opened with 8 partitions" log_id=0aMfkUnG000 service=storage-engine index=tsi
ts=2022-05-09T22:11:49.326507Z lvl=info msg="Reindexing TSM data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=29
ts=2022-05-09T22:11:49.326524Z lvl=info msg="Reindexing WAL data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=29
unexpected fault address 0x7f52ef801a30
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f52ef801a30 pc=0x70c6a1]
goroutine 76630 [running]:
runtime.throw({0x2c15153, 0x0})
/usr/local/go-1.17.6/src/runtime/panic.go:1198 +0x71 fp=0xc0049eafb8 sp=0xc0049eaf88 pc=0x6d7db1
runtime.sigpanic()
/usr/local/go-1.17.6/src/runtime/signal_unix.go:742 +0x2f6 fp=0xc0049eb008 sp=0xc0049eafb8 pc=0x6ee196
runtime.memmove()
/usr/local/go-1.17.6/src/runtime/memmove_amd64.s:191 +0x141 fp=0xc0049eb010 sp=0xc0049eb008 pc=0x70c6a1
runtime.slicebytetostring(0xc0049df200, 0x7f52ef801a30, 0x13)
/usr/local/go-1.17.6/src/runtime/string.go:114 +0x69 fp=0xc0049eb048 sp=0xc0049eb010 pc=0x6f31a9
github.com/influxdata/influxdb/v2/v1/services/storage.(*indexSeriesCursor).Value(0x6e9d33, {0x2c14dc0, 0x1})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/v1/services/storage/series_cursor.go:197 +0x69 fp=0xc0049eb0c0 sp=0xc0049eb048 pc=0x21f6609
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024fb0, 0xc004505d88}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:34 +0x1e7 fp=0xc0049eb0f0 sp=0xc0049eb0c0 pc=0x21940c7
github.com/influxdata/influxdb/v2/storage/reads.evalBinaryExpr(0xc004b4e480, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:42 +0x49 fp=0xc0049eb178 sp=0xc0049eb0f0 pc=0x21941c9
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024ce0, 0xc004b4e480}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:18 +0xf7 fp=0xc0049eb1a8 sp=0xc0049eb178 pc=0x2193fd7
github.com/influxdata/influxdb/v2/storage/reads.evalBinaryExpr(0xc004b4e4e0, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:42 +0x49 fp=0xc0049eb230 sp=0xc0049eb1a8 pc=0x21941c9
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024ce0, 0xc004b4e4e0}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:18 +0xf7 fp=0xc0049eb260 sp=0xc0049eb230 pc=0x2193fd7
github.com/influxdata/influxdb/v2/storage/reads.evalBinaryExpr(0xc004b4e540, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:42 +0x49 fp=0xc0049eb2e8 sp=0xc0049eb260 pc=0x21941c9
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024ce0, 0xc004b4e540}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:18 +0xf7 fp=0xc0049eb318 sp=0xc0049eb2e8 pc=0x2193fd7
github.com/influxdata/influxdb/v2/storage/reads.EvalExprBool(...)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:282
github.com/influxdata/influxdb/v2/v1/services/storage.(*indexSeriesCursor).Next(0xc004ad3b80)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/v1/services/storage/series_cursor.go:171 +0x137 fp=0xc0049eb3c8 sp=0xc0049eb318 pc=0x21f6077
github.com/influxdata/influxdb/v2/storage/reads.(*windowAggregateResultSet).Next(0xc002b1e4b0)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/aggregate_resultset.go:79 +0x5f fp=0xc0049eb510 sp=0xc0049eb3c8 pc=0x217dfff
github.com/influxdata/influxdb/v2/storage/flux.(*windowAggregateIterator).handleRead(0xc004a57c00, 0xc004b035c0, {0x4064590, 0xc002b1e4b0})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/flux/reader.go:734 +0x48e fp=0xc0049ebb38 sp=0xc0049eb510 pc=0x21a23ee
github.com/influxdata/influxdb/v2/storage/flux.(*windowAggregateIterator).Do(0xc004a57c00, 0x2ae3c40)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/flux/reader.go:680 +0x288 fp=0xc0049ebbc8 sp=0xc0049ebb38 pc=0x21a1e28
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc004b342c0, {0x4051c98, 0xc004b4c240}, {0x401a660, 0xc004a57c00}, 0x0)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf fp=0xc0049ebc30 sp=0xc0049ebbc8 pc=0x1cb315f
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readWindowAggregateSource).run(0xc004b342c0, {0x4051c98, 0xc004b4c240})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:310 +0x126 fp=0xc0049ebdd8 sp=0xc0049ebc30 pc=0x1cb4fe6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc004b342c0, {0x4051c98, 0xc004b4c240})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168 fp=0xc0049ebe98 sp=0xc0049ebdd8 pc=0x1cb2d08
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readWindowAggregateSource).Run(0xc004b342c0, {0x4051c98, 0xc004b4c240})
:1 +0x33 fp=0xc0049ebec0 sp=0xc0049ebe98 pc=0x1cb97d3
github.com/influxdata/flux/execute.(*executionState).do.func1({0x4053898, 0xc004b342c0})
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e fp=0xc0049ebfc0 sp=0xc0049ebec0 pc=0x135ecce
github.com/influxdata/flux/execute.(*executionState).do·dwrap·8()
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:296 +0x31 fp=0xc0049ebfe0 sp=0xc0049ebfc0 pc=0x135ea91
runtime.goexit()
/usr/local/go-1.17.6/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0049ebfe8 sp=0xc0049ebfe0 pc=0x70b6e1
created by github.com/influxdata/flux/execute.(*executionState).do
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
goroutine 1 [chan receive, 57 minutes]:
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1()
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/cmd/influxd/launcher/cmd.go:110 +0x329
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000384580, {0xc0005a68a0, 0x2, 0x2})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/kit/cli/viper.go:54 +0x1e
github.com/spf13/cobra.(*Command).execute(0xc000384580, {0xc00021c160, 0x2, 0x2})
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000384580)
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
```
Contributor guide
Research direction
Start with the panic path in v1/services/storage/series_cursor.go, especially indexSeriesCursor.Value and Next, then follow the calls through storage/reads/influxql_eval.go and storage/reads/aggregate_resultset.go. Reproduce the bucket delete/create/write loop and verify that repeated operations no longer panic and instead report failures as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100