apache / apache/ignite

Cluster data went back two months after node rejoin.

Open
#10,611 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

Hello,

I have a three-node Apache Ignite cluster using version `2.14.0` on both persistent and in-memory mode, we had to bring down a node for maintenance two months ago and unfortunately, the issues on that server took about 2 months to be resolved.

Yesterday we were able to bring up the node and we did that by just turning on the server and starting the Apache Ignite `systemd` service. Issuing the `control.sh --baseline` command on the first node indicated that the second node (which was down for two months) is now online, problem is the data on the cluster appears to be reverted back to two months ago as well.

Since the topology version had changed I figure the rebalancing procedure had taken effect and data on the other two nodes would overwrite the data on the second node but after `./control.sh --cache idle_verify' I got something like the below error:

```bash
Consistency checked failed, found conflicts [ CounterConflicts=0, hashConflicts=512]
```

I checked the output and it seemed the partition hash of the second node was different than the others so before doing anything else I decided to remove the second node from the cluster by `./control.sh --baseline remove Ignite02` which worked and `./control.sh --cache idle_verify` didn't show any conflicts anymore, however, the data is still two months old.

I highly doubt that the cluster had overwritten the old data on its own, there is probably something that I've done wrong.

Q: Is there any way to restore the correct data?
Q2: If there is something wrong with what I've done, please tell to avoid future mistakes.

Any help is appreciated.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reported Apache Ignite 2.14.0 node-rejoin sequence, including the systemd startup and the control.sh --baseline and --cache idle_verify commands. Determine whether the stale node data can be restored and document the correct recovery procedure and safeguards for future rejoining; the issue provides no source file or test to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.