Introduce Snapshotting to Optimize CRDT DAG Compaction and Traversal
- Dominant language
- Go
- Stars
- 489
- Forks
- 60
- Avg merge
- 1h 15m
- Merged PRs (30d)
- 1
Description
Hi,
I've been playing with this library and it's great.
However, I'm not happy about the DAG's continuous growth, and I'd like to help.
The current implementation of go-ds-crdt maintains a DAG structure to represent state changes in a datastore. While effective, this approach has limitations:
Performance Overhead: Traversing and replaying large DAGs to compute the current state becomes costly as the DAG grows.
Garbage Collection: Stale or redundant nodes in the DAG are retained unnecessarily, increasing storage requirements.
Recovery Times: Rebuilding state after a crash or restart is slow, as it requires replaying the full DAG history.
Snapshotting introduces a mechanism to address these issues. By creating a periodic snapshot of the DAG, we can:
Compact the state into a "base snapshot."
Limit traversal to only the most recent nodes.
Facilitate faster recovery by replaying transactions only since the last snapshot.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by examining the current DAG representation, traversal, compaction, and recovery paths in go-ds-crdt, then determine the snapshot format and lifecycle. Done means snapshots compact retained state, limit replay to newer changes, and improve recovery without losing correctness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- database, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100