[Bug] A race between RocksDB compaction and periodic Raft snapshot save can commit an invalid snapshot directory that contains `__raft_snapshot_meta` but no `data/`
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 636
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 14
Description
### Bug Type (问题类型)
others (please edit later)
### Before submit
- [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)
### Environment (环境信息)
- Server Version: 1.0.0 (Apache Release Version)
- Backend: RocksDB x nodes, HDD or SSD
- OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
- Data Size: xx vertices, xx edges
### Expected & Actual behavior (期望与实际表现)
## Expected Behavior
- If snapshot save is triggered while compaction is active (`state == doing`), snapshot save should return non-OK to JRaft.
- JRaft should not commit a snapshot point unless snapshot data is actually persisted and files are registered.
- Snapshot directories should not be committed in a meta-only state.
## Actual Behavior
- During compaction, snapshot handler can exit early without writing `data/` (pre-fix path).
- Caller still reports success (`done.run(Status.OK())`) to JRaft.
- JRaft commits snapshot metadata, so snapshot index advances.
- Snapshot directory may contain only `__raft_snapshot_meta`, with missing `data/`.
- On restart/load, snapshot data path is missing, causing load/init failure and retry loops.
## Race Condition Timeline (Sequence of Events)
1.
2. `dbCompaction()` starts for partition `P`.
3. `setState(P, doing)` is set, then `op.compactRange()` begins and blocks.
4. While compaction is running, periodic JRaft snapshot timer fires for `P`.
5. JRaft invokes `PartitionStateMachine.onSnapshotSave(writer, done)`.
6. `PartitionStateMachine` calls `SnapshotHandler.onSnapshotSave(writer)`.
7. In vulnerable behavior, handler exits early on busy state (`state == doing`) without writing `data/` and without registering files.
8. Control returns normally, so caller executes `done.run(Status.OK())`.
9. JRaft treats snapshot save as successful and commits snapshot metadata (`__raft_snapshot_meta`), advancing snapshot index.
10. Compaction later completes and state changes to `compactionDone`.
11. A later blank-task snapshot may be triggered, but this is a new snapshot attempt, not a repair of the already committed bad snapshot.
12. If restart/load occurs before a newer valid snapshot supersedes it, loader targets the committed corrupt snapshot and fails on missing `data/`.## Disk/Log Fingerprint
## Corrupt snapshot signature:
- `snapshot_/__raft_snapshot_meta` exists
- `snapshot_/data/` missing
- `snapshot_/should_not_load` often missing for early-return path
## Typical failure path:
- snapshot save appears successful at callback level
- snapshot load later fails because `snapshot_/data` is absent
## Impact
- Affected partition may become unavailable after restart/snapshot load.
- Partition can enter repeated init failure/retry.
- Operational recovery may require manual cleanup and restart if no newer valid snapshot is available.
## Severity
High (data-plane availability and recovery risk at partition level).
### Vertex/Edge example (问题点 / 边数据举例)
```javascript
```
### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
```javascript
```
Contributor guide
Research direction
Trace PartitionStateMachine.onSnapshotSave(writer, done) into SnapshotHandler.onSnapshotSave(writer), and compare that path with dbCompaction() and op.compactRange(). Reproduce or test the compaction race, then verify that a busy compaction cannot report snapshot success or leave a committed snapshot containing only __raft_snapshot_meta without data/.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100