influxdata / influxdata/influxdb

Wrong data gets persisted when writing same points

Open
#24,388 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hello,

I opened a ticket in the past where I thought data would be lost: [#17735](https://github.com/influxdata/influxdb/issues/17735)

There I also uploaded a simple project to Google Drive where you can reproduce this behaviour.

Now through more investigation and a lot of testing I found out that the data is not lost, instead just wrong data points get persisted.

So how does this happen.
Sometimes it happens in our application that we write points and immediately after that the points get updated. (We already changed the code in our application to avoid this now as good as possible)
And this also only happens if you write a bigger amount of data. (At least with the default configuration of influx)

Here from my observations the things that have to happen, so that this problem occurs:
1. You have to write at least as much data so that the WAL directory is over it's limit and moves the data to the DATA directory (cache-snapshot-memory-size)
2. Then because of this, it has to happen that the DATA directory then contains 8 level one files, so that the automatic compression of the data files start.
3. While this there is still new data written to the WAL directory which again reaches it's limit and moves the data to the DATA directory
4. And to me it seems like if this happens together, that there is data written from the WAL directory to the DATA directory AND the files in the DATA directory get compressed AND there are values for the same data points written in this files, that in this case the wrong values (the first written ones) get persisted instead of the newer ones.

So two ways I found to fix this with configuration:
1. Setting cache-snapshot-memory-size to a high value (1g), so that even if you write a lot of data all data can be stored in the WAL directory and then all of it together is moved to the DATA directory (this is the solution we use now)
2. Set compact-full-write-cold-duration to 1s, so that data in the DATA directory get compressed immediately and it never happens that the compression starts because there are 8 level one files (I am not sure if this would really solve the problem, but at least I couldn't reproduce the behaviour with this setting)

We are still using influx 1.8.
I don't know if this would be still a problem with influx 2 oder 3 then.
If you could confirm that this is not a problem any more in newer influx versions then we would upgrade as fast as possible.

Greetings
Michael

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue on InfluxDB 1.8 using the cache-snapshot-memory-size and compact-full-write-cold-duration settings described in the report. Exercise concurrent WAL-to-DATA snapshots and DATA compaction with updates to the same points; done means the newer values are persisted consistently rather than the first-written values.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.