influxdata / influxdata/influxdb
Loss of data on write(The time stamps are all different)
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
Writing data under high concurrency using a java client loses data, none of the timestamps of this data are repeated
1. A total of 3,000,000 points are written to the data using WriteApi, where WriteOptions is configured as follows:WriteOptions writeOptions = WriteOptions.builder()
.batchSize(5000)
.flushInterval(2000)
.bufferLimit(20000)
.jitterInterval(1000)
.retryInterval(5000)
.build().
2.Then it writes 400 points at a time, sleeps for 200ms, and then continues to cycle through 400 points, sleeps for 200ms...,Until the all point has been written.
__Expected behavior:__
Hopefully this very serious problem will be fixed in version 2.8
__Actual behavior:__
The first time the above steps were performed more than 600 points were lost and the second time more than 4000 points were lost
__Environment info:__
System: CentOS Linux release 7.4.1708 (Core)
CPU: Intel(R) Xeon(R) E5620 @ 2.40GHz * 4 core
Memory: 72G
InfluxDB Server: V2.7.1
__Config:__
bolt-path = "/usr/local/influxdb/data/bolt/influxd.bolt"
engine-path = "/usr/local/influxdb/data/engine"
sqlite-path = "/usr/local/influxdb/data/sqlite/influxd.sqlite"
http-bind-address = "192.168.1.199:8086"
http-idle-timeout = "1m0s"
http-read-timeout = "60s"
http-write-timeout = "60s"
query-concurrency = 70
query-initial-memory-bytes = 10485760
query-memory-bytes = 10485760
query-max-memory-bytes = 1073741824
storage-cache-max-memory-size = 2147483648
storage-compact-throughput-burst = 134217728
storage-max-index-log-file-size = 10485760
flux-log-enabled = "true"
log-level = "info"
```
Contributor guide
Research direction
Start by reproducing the report with the Java WriteApi and the stated WriteOptions against InfluxDB 2.7.1, writing 400-point batches and comparing submitted points with stored points and timestamps. Trace the WriteApi buffering, flushing, and retry behavior involved in the reproduction. Done means the reported concurrent-write scenario retains all points without timestamp-related loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100