influxdata / influxdata/influxdb
Error 500 in some data when writing data over HTTP, writable after restart
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
Query data while writing, guess it's a concurrency related issue, can't happen every time.
1. 50,000 points written per second, guessing is related to cardinality(series).
2. Constantly query the written data.
3. wait, until some data can't be written.
samples:
ms_aaa_request_duration_detail,buz_env=formal,code=200,gameid=aaa,hostname=bbbb,ip=192.168.1.1,project=中文,source=/home/fee/aaa/bbbb/log/general.log,subcode=1,url=authxxx avg=46.0,max=46.0,count=1.0 1665205177446141543
__Expected behavior:__
No error and successfully write data/query data.
__Actual behavior:__
Some data queries timeout or writes timeout
__Environment info:__
* System info: Linux 4.19.0-19-amd64 x86_64
* InfluxDB version: InfluxDB v2.4.0 (git: de247bab08) build_date: 2022-08-18T19:41:15Z
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
__Logs:__
no.
__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```
[goroutine-49-1010.txt](https://github.com/influxdata/influxdb/files/9808664/goroutine-49-1010.txt)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the concurrent HTTP write/query workload described in the issue against InfluxDB v2.4.0, using the provided 50,000-points-per-second scenario. Inspect goroutine-49-1010.txt and collect the listed pprof and iostat outputs while the failure occurs. Done means writes and queries no longer time out under the reported workload.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100