influxdata / influxdata/influxdb

[2.x] [EDR] Replication leaves data gaps behind

Open
#27,567 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/2.x area/replications
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hi everyone,
I am using the replication feature of influx to copy the data from local IPC influx to the AWS influx.
It's great. It's working most of the time. But from time to time, I noticed that, there are some data gaps on the AWS influx, but not local IPC influx.
It's random, from minutes to hours and it might happen with live data or with historical data after internet restoration.
So, the max queue on the local IPC is set to be xGB, which is roughly for 1 week.
And the queue to be synchronized is 0 bytes or a few bytes. And the status is purely 204.

On the IPC side, I could understand it, since it got 204, it removed it from the queue.

On the AWS side, I couldn't. Since it confirmed that the data has arrived, but somehow the data is not written.

What I could think of is
1. Maybe the IPC influx accidentially wrapped an empty message and sent it out and after confirmation of 204, it removed from the local queue. Meanwhile, AWS influx got the empty mesage and wrote it to the disk
2. Maybe nothing wrong on the IPC influx, but the AWS influx just got confused after got the data and failed to write to the disk. (Well, I tried to grep error messages from the log file, but I got none. Yeah, no error recorded.)

__Steps to reproduce:__

I am not sure how you could re-produce this bug. Because both sides (IPC and AWS) haven't recorded any errors. But it has happened to me on 7 IPCs (in total, I have 50+ IPCs).

__Expected behaviour:__
Data got copied from IPC influx to AWS influx without data gap.

__Actual behaviour:__
Some random data gap left behind on AWS influx (not IPC influx).

__Environment info:__

IPC is running with windows. AWS is Amazon Linux (influx is a container). Maybe this is the cause? Influx is V2.8.

Please let me know, if you need more information.

Image

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named, and the issue has no reliable reproduction. Start by tracing the replication queue and HTTP 204 flow between the local IPC instance and AWS, comparing queue state, destination writes, and logs during a reported gap. Done means identifying a reproducible failure or a confirmed cause of the missing data.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.