apache / apache/hugegraph

[Bug] A race between RocksDB compaction and periodic Raft snapshot save can commit an invalid snapshot directory that contains `__raft_snapshot_meta` but no `data/`

Open
#3,162 1 comment 0 reactions 0 assignees View on GitHub
bug raft rocksdb
Dominant language
Java
Stars
3.2k
Forks
636
Avg merge
3d 11h
Merged PRs (30d)
14

Description

### Bug Type (问题类型)

others (please edit later)

### Before submit

- [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

### Environment (环境信息)

- Server Version: 1.0.0 (Apache Release Version)
- Backend: RocksDB x nodes, HDD or SSD
- OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
- Data Size: xx vertices, xx edges

### Expected & Actual behavior (期望与实际表现)

## Expected Behavior

- If snapshot save is triggered while compaction is active (`state == doing`), snapshot save should return non-OK to JRaft.
- JRaft should not commit a snapshot point unless snapshot data is actually persisted and files are registered.
- Snapshot directories should not be committed in a meta-only state.

## Actual Behavior

- During compaction, snapshot handler can exit early without writing `data/` (pre-fix path).
- Caller still reports success (`done.run(Status.OK())`) to JRaft.
- JRaft commits snapshot metadata, so snapshot index advances.
- Snapshot directory may contain only `__raft_snapshot_meta`, with missing `data/`.
- On restart/load, snapshot data path is missing, causing load/init failure and retry loops.

## Race Condition Timeline (Sequence of Events)
1.
2. `dbCompaction()` starts for partition `P`.
3. `setState(P, doing)` is set, then `op.compactRange()` begins and blocks.
4. While compaction is running, periodic JRaft snapshot timer fires for `P`.
5. JRaft invokes `PartitionStateMachine.onSnapshotSave(writer, done)`.
6. `PartitionStateMachine` calls `SnapshotHandler.onSnapshotSave(writer)`.
7. In vulnerable behavior, handler exits early on busy state (`state == doing`) without writing `data/` and without registering files.
8. Control returns normally, so caller executes `done.run(Status.OK())`.
9. JRaft treats snapshot save as successful and commits snapshot metadata (`__raft_snapshot_meta`), advancing snapshot index.
10. Compaction later completes and state changes to `compactionDone`.
11. A later blank-task snapshot may be triggered, but this is a new snapshot attempt, not a repair of the already committed bad snapshot.
12. If restart/load occurs before a newer valid snapshot supersedes it, loader targets the committed corrupt snapshot and fails on missing `data/`.## Disk/Log Fingerprint

## Corrupt snapshot signature:
- `snapshot_/__raft_snapshot_meta` exists
- `snapshot_/data/` missing
- `snapshot_/should_not_load` often missing for early-return path

## Typical failure path:
- snapshot save appears successful at callback level
- snapshot load later fails because `snapshot_/data` is absent

## Impact

- Affected partition may become unavailable after restart/snapshot load.
- Partition can enter repeated init failure/retry.
- Operational recovery may require manual cleanup and restart if no newer valid snapshot is available.

## Severity
High (data-plane availability and recovery risk at partition level).

### Vertex/Edge example (问题点 / 边数据举例)

```javascript

```

### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

```javascript

```

Contributor guide

Open the contributing guide

Research direction

Trace PartitionStateMachine.onSnapshotSave(writer, done) into SnapshotHandler.onSnapshotSave(writer), and compare that path with dbCompaction() and op.compactRange(). Reproduce or test the compaction race, then verify that a busy compaction cannot report snapshot success or leave a committed snapshot containing only __raft_snapshot_meta without data/.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.