influxdata / influxdata/influxdb
Wrong data gets persisted when writing same points
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Hello,
I opened a ticket in the past where I thought data would be lost: [#17735](https://github.com/influxdata/influxdb/issues/17735)
There I also uploaded a simple project to Google Drive where you can reproduce this behaviour.
Now through more investigation and a lot of testing I found out that the data is not lost, instead just wrong data points get persisted.
So how does this happen.
Sometimes it happens in our application that we write points and immediately after that the points get updated. (We already changed the code in our application to avoid this now as good as possible)
And this also only happens if you write a bigger amount of data. (At least with the default configuration of influx)
Here from my observations the things that have to happen, so that this problem occurs:
1. You have to write at least as much data so that the WAL directory is over it's limit and moves the data to the DATA directory (cache-snapshot-memory-size)
2. Then because of this, it has to happen that the DATA directory then contains 8 level one files, so that the automatic compression of the data files start.
3. While this there is still new data written to the WAL directory which again reaches it's limit and moves the data to the DATA directory
4. And to me it seems like if this happens together, that there is data written from the WAL directory to the DATA directory AND the files in the DATA directory get compressed AND there are values for the same data points written in this files, that in this case the wrong values (the first written ones) get persisted instead of the newer ones.
So two ways I found to fix this with configuration:
1. Setting cache-snapshot-memory-size to a high value (1g), so that even if you write a lot of data all data can be stored in the WAL directory and then all of it together is moved to the DATA directory (this is the solution we use now)
2. Set compact-full-write-cold-duration to 1s, so that data in the DATA directory get compressed immediately and it never happens that the compression starts because there are 8 level one files (I am not sure if this would really solve the problem, but at least I couldn't reproduce the behaviour with this setting)
We are still using influx 1.8.
I don't know if this would be still a problem with influx 2 oder 3 then.
If you could confirm that this is not a problem any more in newer influx versions then we would upgrade as fast as possible.
Greetings
Michael
Contributor guide
Research direction
Start by reproducing the issue on InfluxDB 1.8 using the cache-snapshot-memory-size and compact-full-write-cold-duration settings described in the report. Exercise concurrent WAL-to-DATA snapshots and DATA compaction with updates to the same points; done means the newer values are persisted consistently rather than the first-written values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100