influxdata / influxdata/influxdb
[2.x] panic: invalid memory address or nil pointer dereference, (*integerGroupTable).advanceCursor
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Steps to reproduce:
List the minimal actions needed to reproduce the behavior.
- Single threaded python script to insert datapoints in a tight loop, using async io writes
- Query to count the number of datapoints in past few minutes
query: f'from(bucket: "{bucket}") |> range(start: -10m) |> group() |> window(every: 1s) |> count()'
Expected behavior:
inserts to complete, query to return
Actual behavior:
influxd hits an error (see below)
I have to kill the process and start it again.
Environment info:
-
System info: Run
uname -srmand copy the output here
Darwin 21.6.0 x86_64 -
InfluxDB version: Run
influxd versionand copy the output here
InfluxDB 2.3.0 (git: 090f681737) build_date: 2022-06-16T18:48:06Z
all datapoints are test data:
single measurement, 10 tags, 15 fields (all integers). Not high cardinality (series count probably in the low thousands)
5000 datapoints per write
Config:
Copy any non-default config values here or attach the full config as a gist or file.
Default config, no modifications.
Logs:
2023-02-04T03:49:30.271055Z info Dispatcher panic {"log_id": "0fnCphPG000", "service": "storage-reads", "component": "dispatcher", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 133856 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*poolDispatcher).recover(0xc000b3afc0)
github.com/influxdata/flux@v0.171.0/execute/recover.go:53 +0x205
panic({0x608d980, 0xd120330})
runtime/panic.go:838 +0x207
github.com/influxdata/influxdb/v2/storage/flux.(*integerGroupTable).advanceCursor(0xc0054022a0)
github.com/influxdata/influxdb/v2/storage/flux/table.gen.go:1957 +0xc0
github.com/influxdata/influxdb/v2/storage/flux.(*integerGroupTable).advance(0xc0054022a0)
github.com/influxdata/influxdb/v2/storage/flux/table.gen.go:1710 +0x85
github.com/influxdata/influxdb/v2/storage/flux.(*table).do(0xc0045be820?, 0xc046a3a870, 0xc05924d810)
github.com/influxdata/influxdb/v2/storage/flux/table.go:95 +0x213
github.com/influxdata/influxdb/v2/storage/flux.(*integerGroupTable).Do(0x6331300?, 0xc0045be801?)
github.com/influxdata/influxdb/v2/storage/flux/table.gen.go:1690 +0x37
github.com/influxdata/flux/execute.(*consecutiveTransportTable).Do(0xc046a3a840, 0xc00a9a0180)
github.com/influxdata/flux@v0.171.0/execute/transport.go:549 +0x87
github.com/influxdata/flux/stdlib/universe.(*fixedWindowTransformation).Process(0xc0040d4a80, {0xb1, 0xc6, 0xa8, 0x26, 0x2f, 0x15, 0x57, 0x70, 0x99, ...}, ...)
github.com/influxdata/flux@v0.171.0/stdlib/universe/window.go:356 +0xa3f
github.com/influxdata/flux/execute.(*transformationTransportAdapter).ProcessMessage(0xc061522b40, {0x68c9fd8, 0xc001ad10e0})
github.com/influxdata/flux@v0.171.0/execute/transport.go:647 +0x195
github.com/influxdata/flux/execute.(*consecutiveTransport).processMessage(0xc008dc63c0, {0x68c9fd8, 0xc001ad10e0})
github.com/influxdata/flux@v0.171.0/execute/transport.go:292 +0xca
github.com/influxdata/flux/execute.(*consecutiveTransport).processMessages(0xc008dc63c0, {0x68c9210?, 0xc0614ff000?}, 0xa)
github.com/influxdata/flux@v0.171.0/execute/transport.go:246 +0xe5
github.com/influxdata/flux/execute.(*poolDispatcher).doWork(0xc000b3afc0, {0x68c9210, 0xc0614ff000})
github.com/influxdata/flux@v0.171.0/execute/dispatcher.go:157 +0x19f
github.com/influxdata/flux/execute.(*poolDispatcher).run(0xc000b3afc0, {0x68c9210, 0xc0614ff000})
github.com/influxdata/flux@v0.171.0/execute/dispatcher.go:135 +0x4f
github.com/influxdata/flux/execute.(*poolDispatcher).Start.func1()
github.com/influxdata/flux@v0.171.0/execute/dispatcher.go:75 +0x86
created by github.com/influxdata/flux/execute.(*poolDispatcher).Start
github.com/influxdata/flux@v0.171.0/execute/dispatcher.go:71 +0x48
2023-02-04T03:49:30.274949Z warn internal error not returned to client {"log_id": "0fnCphPG000", "handler": "error_logger", "error": "context canceled"}
Performance:
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with storage/flux/table.gen.go at integerGroupTable.advanceCursor and read the surrounding advance path in storage/flux/table.go. Reproduce the issue using concurrent writes and the grouped, windowed count query described in the report, then inspect the panic path. Done means the query returns without an influxd panic while inserts continue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100