influxdata / influxdata/influxdb

unexpected fault address 0x7fe1aa763040

Open
#21,759 0 comments 0 reactions 0 assignees View on GitHub
area/docker kind/bug os/windows panic
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__

I have a minimal InfluxDB + Telegraf + MQTT docker-compose stack running on Windows. I noticed that InfluxDB is crashing randomly with an `unexpected fault address 0x7fe1aa763040` and a huge stacktrace. The crashes are random but they seem to happen every day.

__Environment info:__

* System info: Docker on Windows 10
* InfluxDB version: Docker image influxdb:2.0.7

__Config:__
This is docker-compose config just in case it's relevant:

```
influxdb:
image: influxdb:2.0.7
ports:
- 8086:8086
volumes:
- ./influxdb:/var/lib/influxdb2
```

__Logs:__

Full logs can be found in here: https://textbin.net/lpdrqhehxo

```
unexpected fault address 0x7fe1aa763040
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fe1aa763040 pc=0xfdad60]

goroutine 1311 [running]:
runtime.throw(0x21cdf84, 0x5)
/usr/local/go/src/runtime/panic.go:1116 +0x74 fp=0xc000b427a0 sp=0xc000b42770 pc=0x43db14
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:717 +0x485 fp=0xc000b427d0 sp=0xc000b427a0 pc=0x4546a5
go.etcd.io/bbolt.(*DB).meta(0xc0000b0e00, 0xc001702040)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:901 +0x40 fp=0xc000b42820 sp=0xc000b427d0 pc=0xfdad60
go.etcd.io/bbolt.(*Tx).init(0xc001c0a000, 0xc0000b0e00)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/tx.go:50 +0x90 fp=0xc000b42888 sp=0xc000b42820 pc=0xfe3f70
go.etcd.io/bbolt.(*DB).beginTx(0xc0000b0e00, 0x21cc90e, 0x4, 0x2)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:567 +0x12a fp=0xc000b428f8 sp=0xc000b42888 pc=0xfd958a
go.etcd.io/bbolt.(*DB).Begin(0xc0000b0e00, 0x0, 0x0, 0xc000b42980, 0x101ace5)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:544 +0x70 fp=0xc000b42928 sp=0xc000b428f8 pc=0xfd9410
go.etcd.io/bbolt.(*DB).View(0xc0000b0e00, 0xc000b429e0, 0x0, 0x0)
/home/circleci/go/pkg/mod/go.etcd.io/bbolt@v1.3.5/db.go:709 +0x4b fp=0xc000b42990 sp=0xc000b42928 pc=0xfda0cb
github.com/influxdata/influxdb/v2/bolt.(*KVStore).View(0xc000a0d840, 0x3dc22c0, 0xc0024e82a0, 0xc000ab9130, 0x0, 0x0)
/home/circleci/go/src/github.com/influxdata/influxdb/bolt/kv.go:154 +0x117 fp=0xc000b42a20 sp=0xc000b42990 pc=0x101b137
github.com/influxdata/influxdb/v2/kv.(*Service).ListTargets(0xc000a0b130, 0x3dc22c0, 0xc0024e80c0, 0x0, 0x0, 0x0, 0x0, 0x3dc22c0, 0xc0024e80c0, 0x20, ...)
/home/circleci/go/src/github.com/influxdata/influxdb/kv/scrapers.go:107 +0x112 fp=0xc000b42a80 sp=0xc000b42a20 pc=0xff62f2
github.com/influxdata/influxdb/v2/gather.(*Scheduler).doGather(0xc0010d82c0, 0x3dc2200, 0xc000a0d240)
/home/circleci/go/src/github.com/influxdata/influxdb/gather/scheduler.go:110 +0x125 fp=0xc000b42d48 sp=0xc000b42a80 pc=0x1c5c005
github.com/influxdata/influxdb/v2/gather.(*Scheduler).run(0xc0010d82c0, 0x3dc2200, 0xc000a0d240, 0x3dc2200, 0xc000a0d240)
/home/circleci/go/src/github.com/influxdata/influxdb/gather/scheduler.go:99 +0x5e fp=0xc000b42dd8 sp=0xc000b42d48 pc=0x1c5bdfe
github.com/influxdata/influxdb/v2/gather.(*Scheduler).Run(0xc0010d82c0, 0x3dc2200, 0xc000a0d240, 0x1, 0xc001834720)
/home/circleci/go/src/github.com/influxdata/influxdb/gather/scheduler.go:90 +0x7b fp=0xc000b42e10 sp=0xc000b42dd8 pc=0x1c5bd5b
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run.func3(0xc000953260, 0xc0010d82c0, 0x3dc2200, 0xc000a0d240, 0xc001834720)
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:611 +0x1dd fp=0xc000b42fb8 sp=0xc000b42e10 pc=0x1ee147d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000b42fc0 sp=0xc000b42fb8 pc=0x476ea1
created by github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:608 +0x6912

goroutine 1 [chan receive, 2060 minutes]:
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1(0x0, 0x0)
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/cmd.go:109 +0x352
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000acc840, 0xa2658c8, 0x0, 0x0, 0x0, 0x0)
/home/circleci/go/src/github.com/influxdata/influxdb/kit/cli/viper.go:54 +0x2b
github.com/spf13/cobra.(*Command).execute(0xc000acc840, 0xc0001921c0, 0x0, 0x0, 0xc000acc840, 0xc0001921c0)
/home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000acc840, 0xc000031f48, 0x1, 0x1)
/home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/home/circleci/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:53 +0x39b
```

Contributor guide

Open the contributing guide

Research direction

Start with the reported stack trace in go.etcd.io/bbolt/db.go and the InfluxDB paths bolt/kv.go, kv/scrapers.go, and gather/scheduler.go. Reproduce the crash using the provided Docker Compose setup and logs, then determine the cause and verify that the database no longer exits unexpectedly under the same workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.