influxdata / influxdata/influxdb

Error 500 in some data when writing data over HTTP, writable after restart

Open
#23,816 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
Query data while writing, guess it's a concurrency related issue, can't happen every time.

1. 50,000 points written per second, guessing is related to cardinality(series).
2. Constantly query the written data.
3. wait, until some data can't be written.

samples:
ms_aaa_request_duration_detail,buz_env=formal,code=200,gameid=aaa,hostname=bbbb,ip=192.168.1.1,project=中文,source=/home/fee/aaa/bbbb/log/general.log,subcode=1,url=authxxx avg=46.0,max=46.0,count=1.0 1665205177446141543

__Expected behavior:__
No error and successfully write data/query data.

__Actual behavior:__
Some data queries timeout or writes timeout

__Environment info:__

* System info: Linux 4.19.0-19-amd64 x86_64
* InfluxDB version: InfluxDB v2.4.0 (git: de247bab08) build_date: 2022-08-18T19:41:15Z

__Config:__
Copy any non-default config values here or attach the full config as a gist or file.

__Logs:__
no.

__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```
[goroutine-49-1010.txt](https://github.com/influxdata/influxdb/files/9808664/goroutine-49-1010.txt)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the concurrent HTTP write/query workload described in the issue against InfluxDB v2.4.0, using the provided 50,000-points-per-second scenario. Inspect goroutine-49-1010.txt and collect the listed pprof and iostat outputs while the failure occurs. Done means writes and queries no longer time out under the reported workload.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.