influxdata / influxdata/influxdb

Bucket delete, create, write causes occasional panic

Open
#23,333 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I am using influxdb 2.1.1 in a kubernetes pod. We have a script that deletes a bucket, creates it, and writes configuration values to it (synchronously). Occasionally, this script was getting `('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))` error codes. Adding retry logic, we started seeing `111: Err connection refused` errors. This connection refused was due to the pod panicing, and this panic behavior was reproduced manually as well.

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Start influxdb pod
2. Write values to bucket from tar.gz point-protocol file
3. Drop bucket
4. Create bucket
5. Repeat steps 2-4 until panic happens

__Expected behavior:__
No panic should occur. Bucket create/delete and writing from file should report if it failed, but not cause pod panic.

__Actual behavior:__
The pod will panic occasionally. This happens during test teardown once or twice out of 12 times. While doing this process manually, it took about 8 times to trigger.

__Environment info:__

* Linux 4.15.0-163-generic x86_64
* InfluxDB v2.1.1 (git: 657e1839) build_date: 2022-05-10T14:36:50Z
* Kubernetes pod

__Config:__
```
influx config list --json
{}
```

__Logs:__
```
ts=2022-05-09T22:10:56.342751Z lvl=info msg="index opened with 8 partitions" log_id=0aMfkUnG000 service=storage-engine index=tsi
ts=2022-05-09T22:10:56.343895Z lvl=info msg="Reindexing TSM data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=28
ts=2022-05-09T22:10:56.343915Z lvl=info msg="Reindexing WAL data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=28
ts=2022-05-09T22:11:49.325626Z lvl=info msg="index opened with 8 partitions" log_id=0aMfkUnG000 service=storage-engine index=tsi
ts=2022-05-09T22:11:49.326507Z lvl=info msg="Reindexing TSM data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=29
ts=2022-05-09T22:11:49.326524Z lvl=info msg="Reindexing WAL data" log_id=0aMfkUnG000 service=storage-engine engine=tsm1 db_shard_id=29
unexpected fault address 0x7f52ef801a30
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f52ef801a30 pc=0x70c6a1]
goroutine 76630 [running]:
runtime.throw({0x2c15153, 0x0})
/usr/local/go-1.17.6/src/runtime/panic.go:1198 +0x71 fp=0xc0049eafb8 sp=0xc0049eaf88 pc=0x6d7db1
runtime.sigpanic()
/usr/local/go-1.17.6/src/runtime/signal_unix.go:742 +0x2f6 fp=0xc0049eb008 sp=0xc0049eafb8 pc=0x6ee196
runtime.memmove()
/usr/local/go-1.17.6/src/runtime/memmove_amd64.s:191 +0x141 fp=0xc0049eb010 sp=0xc0049eb008 pc=0x70c6a1
runtime.slicebytetostring(0xc0049df200, 0x7f52ef801a30, 0x13)
/usr/local/go-1.17.6/src/runtime/string.go:114 +0x69 fp=0xc0049eb048 sp=0xc0049eb010 pc=0x6f31a9
github.com/influxdata/influxdb/v2/v1/services/storage.(*indexSeriesCursor).Value(0x6e9d33, {0x2c14dc0, 0x1})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/v1/services/storage/series_cursor.go:197 +0x69 fp=0xc0049eb0c0 sp=0xc0049eb048 pc=0x21f6609
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024fb0, 0xc004505d88}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:34 +0x1e7 fp=0xc0049eb0f0 sp=0xc0049eb0c0 pc=0x21940c7
github.com/influxdata/influxdb/v2/storage/reads.evalBinaryExpr(0xc004b4e480, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:42 +0x49 fp=0xc0049eb178 sp=0xc0049eb0f0 pc=0x21941c9
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024ce0, 0xc004b4e480}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:18 +0xf7 fp=0xc0049eb1a8 sp=0xc0049eb178 pc=0x2193fd7
github.com/influxdata/influxdb/v2/storage/reads.evalBinaryExpr(0xc004b4e4e0, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:42 +0x49 fp=0xc0049eb230 sp=0xc0049eb1a8 pc=0x21941c9
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024ce0, 0xc004b4e4e0}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:18 +0xf7 fp=0xc0049eb260 sp=0xc0049eb230 pc=0x2193fd7
github.com/influxdata/influxdb/v2/storage/reads.evalBinaryExpr(0xc004b4e540, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:42 +0x49 fp=0xc0049eb2e8 sp=0xc0049eb260 pc=0x21941c9
github.com/influxdata/influxdb/v2/storage/reads.evalExpr({0x4024ce0, 0xc004b4e540}, {0x3ffdc20, 0xc004ad3b80})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:18 +0xf7 fp=0xc0049eb318 sp=0xc0049eb2e8 pc=0x2193fd7
github.com/influxdata/influxdb/v2/storage/reads.EvalExprBool(...)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/influxql_eval.go:282
github.com/influxdata/influxdb/v2/v1/services/storage.(*indexSeriesCursor).Next(0xc004ad3b80)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/v1/services/storage/series_cursor.go:171 +0x137 fp=0xc0049eb3c8 sp=0xc0049eb318 pc=0x21f6077
github.com/influxdata/influxdb/v2/storage/reads.(*windowAggregateResultSet).Next(0xc002b1e4b0)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/reads/aggregate_resultset.go:79 +0x5f fp=0xc0049eb510 sp=0xc0049eb3c8 pc=0x217dfff
github.com/influxdata/influxdb/v2/storage/flux.(*windowAggregateIterator).handleRead(0xc004a57c00, 0xc004b035c0, {0x4064590, 0xc002b1e4b0})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/flux/reader.go:734 +0x48e fp=0xc0049ebb38 sp=0xc0049eb510 pc=0x21a23ee
github.com/influxdata/influxdb/v2/storage/flux.(*windowAggregateIterator).Do(0xc004a57c00, 0x2ae3c40)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/storage/flux/reader.go:680 +0x288 fp=0xc0049ebbc8 sp=0xc0049ebb38 pc=0x21a1e28
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc004b342c0, {0x4051c98, 0xc004b4c240}, {0x401a660, 0xc004a57c00}, 0x0)
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf fp=0xc0049ebc30 sp=0xc0049ebbc8 pc=0x1cb315f
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readWindowAggregateSource).run(0xc004b342c0, {0x4051c98, 0xc004b4c240})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:310 +0x126 fp=0xc0049ebdd8 sp=0xc0049ebc30 pc=0x1cb4fe6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc004b342c0, {0x4051c98, 0xc004b4c240})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168 fp=0xc0049ebe98 sp=0xc0049ebdd8 pc=0x1cb2d08
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readWindowAggregateSource).Run(0xc004b342c0, {0x4051c98, 0xc004b4c240})
:1 +0x33 fp=0xc0049ebec0 sp=0xc0049ebe98 pc=0x1cb97d3
github.com/influxdata/flux/execute.(*executionState).do.func1({0x4053898, 0xc004b342c0})
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e fp=0xc0049ebfc0 sp=0xc0049ebec0 pc=0x135ecce
github.com/influxdata/flux/execute.(*executionState).do·dwrap·8()
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:296 +0x31 fp=0xc0049ebfe0 sp=0xc0049ebfc0 pc=0x135ea91
runtime.goexit()
/usr/local/go-1.17.6/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0049ebfe8 sp=0xc0049ebfe0 pc=0x70b6e1
created by github.com/influxdata/flux/execute.(*executionState).do
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
goroutine 1 [chan receive, 57 minutes]:
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1()
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/cmd/influxd/launcher/cmd.go:110 +0x329
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000384580, {0xc0005a68a0, 0x2, 0x2})
/bitnami/blacksmith-sandox/influxdb-2.1.1/src/github.com/influxdata/influxdb/kit/cli/viper.go:54 +0x1e
github.com/spf13/cobra.(*Command).execute(0xc000384580, {0xc00021c160, 0x2, 0x2})
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000384580)
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/bitnami/blacksmith-sandox/influxdb-2.1.1/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
```

Contributor guide

Open the contributing guide

Research direction

Start with the panic path in v1/services/storage/series_cursor.go, especially indexSeriesCursor.Value and Next, then follow the calls through storage/reads/influxql_eval.go and storage/reads/aggregate_resultset.go. Reproduce the bucket delete/create/write loop and verify that repeated operations no longer panic and instead report failures as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.