influxdata / influxdata/influxdb
unexpected fault address 0x7fe1aa763040
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
I have a minimal InfluxDB + Telegraf + MQTT docker-compose stack running on Windows. I noticed that InfluxDB is crashing randomly with an `unexpected fault address 0x7fe1aa763040` and a huge stacktrace. The crashes are random but they seem to happen every day.
__Environment info:__
* System info: Docker on Windows 10
* InfluxDB version: Docker image influxdb:2.0.7
__Config:__
This is docker-compose config just in case it's relevant:
```
influxdb:
image: influxdb:2.0.7
ports:
- 8086:8086
volumes:
- ./influxdb:/var/lib/influxdb2
```
__Logs:__
Full logs can be found in here: https://textbin.net/lpdrqhehxo
```
unexpected fault address 0x7fe1aa763040
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fe1aa763040 pc=0xfdad60]
goroutine 1311 [running]:
runtime.throw(0x21cdf84, 0x5)
/usr/local/go/src/runtime/panic.go:1116 +0x74 fp=0xc000b427a0 sp=0xc000b42770 pc=0x43db14
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:717 +0x485 fp=0xc000b427d0 sp=0xc000b427a0 pc=0x4546a5
go.etcd.io/bbolt.(*DB).meta(0xc0000b0e00, 0xc001702040)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:901 +0x40 fp=0xc000b42820 sp=0xc000b427d0 pc=0xfdad60
go.etcd.io/bbolt.(*Tx).init(0xc001c0a000, 0xc0000b0e00)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/tx.go:50 +0x90 fp=0xc000b42888 sp=0xc000b42820 pc=0xfe3f70
go.etcd.io/bbolt.(*DB).beginTx(0xc0000b0e00, 0x21cc90e, 0x4, 0x2)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:567 +0x12a fp=0xc000b428f8 sp=0xc000b42888 pc=0xfd958a
go.etcd.io/bbolt.(*DB).Begin(0xc0000b0e00, 0x0, 0x0, 0xc000b42980, 0x101ace5)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:544 +0x70 fp=0xc000b42928 sp=0xc000b428f8 pc=0xfd9410
go.etcd.io/bbolt.(*DB).View(0xc0000b0e00, 0xc000b429e0, 0x0, 0x0)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:709 +0x4b fp=0xc000b42990 sp=0xc000b42928 pc=0xfda0cb
github.com/influxdata/influxdb/v2/bolt.(*KVStore).View(0xc000a0d840, 0x3dc22c0, 0xc0024e82a0, 0xc000ab9130, 0x0, 0x0)
/home/circleci/go/src/github.com/influxdata/influxdb/bolt/kv.go:154 +0x117 fp=0xc000b42a20 sp=0xc000b42990 pc=0x101b137
github.com/influxdata/influxdb/v2/kv.(*Service).ListTargets(0xc000a0b130, 0x3dc22c0, 0xc0024e80c0, 0x0, 0x0, 0x0, 0x0, 0x3dc22c0, 0xc0024e80c0, 0x20, ...)
/home/circleci/go/src/github.com/influxdata/influxdb/kv/scrapers.go:107 +0x112 fp=0xc000b42a80 sp=0xc000b42a20 pc=0xff62f2
github.com/influxdata/influxdb/v2/gather.(*Scheduler).doGather(0xc0010d82c0, 0x3dc2200, 0xc000a0d240)
/home/circleci/go/src/github.com/influxdata/influxdb/gather/scheduler.go:110 +0x125 fp=0xc000b42d48 sp=0xc000b42a80 pc=0x1c5c005
github.com/influxdata/influxdb/v2/gather.(*Scheduler).run(0xc0010d82c0, 0x3dc2200, 0xc000a0d240, 0x3dc2200, 0xc000a0d240)
/home/circleci/go/src/github.com/influxdata/influxdb/gather/scheduler.go:99 +0x5e fp=0xc000b42dd8 sp=0xc000b42d48 pc=0x1c5bdfe
github.com/influxdata/influxdb/v2/gather.(*Scheduler).Run(0xc0010d82c0, 0x3dc2200, 0xc000a0d240, 0x1, 0xc001834720)
/home/circleci/go/src/github.com/influxdata/influxdb/gather/scheduler.go:90 +0x7b fp=0xc000b42e10 sp=0xc000b42dd8 pc=0x1c5bd5b
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run.func3(0xc000953260, 0xc0010d82c0, 0x3dc2200, 0xc000a0d240, 0xc001834720)
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:611 +0x1dd fp=0xc000b42fb8 sp=0xc000b42e10 pc=0x1ee147d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000b42fc0 sp=0xc000b42fb8 pc=0x476ea1
created by github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:608 +0x6912
goroutine 1 [chan receive, 2060 minutes]:
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1(0x0, 0x0)
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/cmd.go:109 +0x352
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000acc840, 0xa2658c8, 0x0, 0x0, 0x0, 0x0)
/home/circleci/go/src/github.com/influxdata/influxdb/kit/cli/viper.go:54 +0x2b
github.com/spf13/cobra.(*Command).execute(0xc000acc840, 0xc0001921c0, 0x0, 0x0, 0xc000acc840, 0xc0001921c0)
/home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000acc840, 0xc000031f48, 0x1, 0x1)
/home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:53 +0x39b
```
Contributor guide
Research direction
Start with the reported stack trace in go.etcd.io/bbolt/db.go and the InfluxDB paths bolt/kv.go, kv/scrapers.go, and gather/scheduler.go. Reproduce the crash using the provided Docker Compose setup and logs, then determine the cause and verify that the database no longer exits unexpectedly under the same workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100