influxdata / influxdata/influxdb
fatal error: concurrent map read and map write with opentsdb?
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. I don't know how to reproduce the issue consistently. It appears to be random.
__Expected behavior:__
No crash
__Actual behavior:__
InfluxDB crashes about once a week (but recovers automatically).
__Environment info:__
There are two influxdb instances running on two identical ppc64le nodes.
Collectd sends data from multiple hosts via the `write_opentsdb` plugin to both instances.
* System info: `Linux 4.14.0-115.14.1.el7a.ppc64le ppc64le`, `RHEL 7.6 for ppc64le`
* InfluxDB version: `InfluxDB v1.7.10 (git: heads/v1.7.10 f46f63d4e2d9684a2dd716594ab609ccd32f0a5b)`
* Other relevant environment details: Bare metal, built with Go `1.12.17` and:
```
go get github.com/influxdata/influxdb
cd $GOPATH/src/github.com/influxdata/influxdb
git checkout tags/v1.7.10 -b v1.7.10
python build.py --package --release --clean --update
```
__Config:__
Non default config entries:
```
reporting-disabled = true
[data]
index-version = "tsi1"
query-log-enabled = false
series-id-set-cache-size = 100
[http]
auth-enabled = true
log-enabled = false
[[opentsdb]]
enabled = true
bind-address = ":4242"
database = "esmon_database"
[continuous_queries]
log-enabled = false
```
__Logs:__
[influxdb_crash_node1.log](https://github.com/influxdata/influxdb/files/4343088/influxdb_crash_node1.log)
[influxdb_crash_node2.log](https://github.com/influxdata/influxdb/files/4343089/influxdb_crash_node2.log)
Contributor guide
Research direction
Start by reviewing influxdb_crash_node1.log and influxdb_crash_node2.log, then trace the enabled [[opentsdb]] input and its concurrent write path. Done means identifying the race and confirming that concurrent OpenTSDB ingestion no longer crashes InfluxDB.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100