influxdata / influxdata/influxdb

Loss of data on write(The time stamps are all different)

Open
#24,352 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
Writing data under high concurrency using a java client loses data, none of the timestamps of this data are repeated

1. A total of 3,000,000 points are written to the data using WriteApi, where WriteOptions is configured as follows:WriteOptions writeOptions = WriteOptions.builder()
.batchSize(5000)
.flushInterval(2000)
.bufferLimit(20000)
.jitterInterval(1000)
.retryInterval(5000)
.build().
2.Then it writes 400 points at a time, sleeps for 200ms, and then continues to cycle through 400 points, sleeps for 200ms...,Until the all point has been written.

__Expected behavior:__
Hopefully this very serious problem will be fixed in version 2.8

__Actual behavior:__
The first time the above steps were performed more than 600 points were lost and the second time more than 4000 points were lost

__Environment info:__
System: CentOS Linux release 7.4.1708 (Core)
CPU: Intel(R) Xeon(R) E5620 @ 2.40GHz * 4 core
Memory: 72G
InfluxDB Server: V2.7.1

__Config:__
bolt-path = "/usr/local/influxdb/data/bolt/influxd.bolt"
engine-path = "/usr/local/influxdb/data/engine"
sqlite-path = "/usr/local/influxdb/data/sqlite/influxd.sqlite"
http-bind-address = "192.168.1.199:8086"
http-idle-timeout = "1m0s"
http-read-timeout = "60s"
http-write-timeout = "60s"
query-concurrency = 70
query-initial-memory-bytes = 10485760
query-memory-bytes = 10485760
query-max-memory-bytes = 1073741824
storage-cache-max-memory-size = 2147483648
storage-compact-throughput-burst = 134217728
storage-max-index-log-file-size = 10485760

flux-log-enabled = "true"
log-level = "info"

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the report with the Java WriteApi and the stated WriteOptions against InfluxDB 2.7.1, writing 400-point batches and comparing submitted points with stored points and timestamps. Trace the WriteApi buffering, flushing, and retry behavior involved in the reproduction. Done means the reported concurrent-write scenario retains all points without timestamp-related loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.