influxdata / influxdata/influxdb

fatal error: concurrent map read and map write with opentsdb?

Open
#17,308 1 comment 0 reactions 0 assignees View on GitHub
1.x area/tsm kind/bug
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. I don't know how to reproduce the issue consistently. It appears to be random.

__Expected behavior:__
No crash

__Actual behavior:__
InfluxDB crashes about once a week (but recovers automatically).

__Environment info:__
There are two influxdb instances running on two identical ppc64le nodes.
Collectd sends data from multiple hosts via the `write_opentsdb` plugin to both instances.
* System info: `Linux 4.14.0-115.14.1.el7a.ppc64le ppc64le`, `RHEL 7.6 for ppc64le`
* InfluxDB version: `InfluxDB v1.7.10 (git: heads/v1.7.10 f46f63d4e2d9684a2dd716594ab609ccd32f0a5b)`
* Other relevant environment details: Bare metal, built with Go `1.12.17` and:
```
go get github.com/influxdata/influxdb
cd $GOPATH/src/github.com/influxdata/influxdb
git checkout tags/v1.7.10 -b v1.7.10

python build.py --package --release --clean --update
```
__Config:__
Non default config entries:
```
reporting-disabled = true

[data]
index-version = "tsi1"
query-log-enabled = false
series-id-set-cache-size = 100

[http]
auth-enabled = true
log-enabled = false

[[opentsdb]]
enabled = true
bind-address = ":4242"
database = "esmon_database"

[continuous_queries]
log-enabled = false
```

__Logs:__
[influxdb_crash_node1.log](https://github.com/influxdata/influxdb/files/4343088/influxdb_crash_node1.log)
[influxdb_crash_node2.log](https://github.com/influxdata/influxdb/files/4343089/influxdb_crash_node2.log)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing influxdb_crash_node1.log and influxdb_crash_node2.log, then trace the enabled [[opentsdb]] input and its concurrent write path. Done means identifying the race and confirming that concurrent OpenTSDB ingestion no longer crashes InfluxDB.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.