ipfs / ipfs/go-ds-crdt

Introduce Snapshotting to Optimize CRDT DAG Compaction and Traversal

Open
#249 3 comments 4 reactions 0 assignees View on GitHub
kind/discussion
Dominant language
Go
Stars
489
Forks
60
Avg merge
1h 15m
Merged PRs (30d)
1

Description

Hi,
I've been playing with this library and it's great.
However, I'm not happy about the DAG's continuous growth, and I'd like to help.

The current implementation of go-ds-crdt maintains a DAG structure to represent state changes in a datastore. While effective, this approach has limitations:

Performance Overhead: Traversing and replaying large DAGs to compute the current state becomes costly as the DAG grows.
Garbage Collection: Stale or redundant nodes in the DAG are retained unnecessarily, increasing storage requirements.
Recovery Times: Rebuilding state after a crash or restart is slow, as it requires replaying the full DAG history.
Snapshotting introduces a mechanism to address these issues. By creating a periodic snapshot of the DAG, we can:

Compact the state into a "base snapshot."
Limit traversal to only the most recent nodes.
Facilitate faster recovery by replaying transactions only since the last snapshot.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by examining the current DAG representation, traversal, compaction, and recovery paths in go-ds-crdt, then determine the snapshot format and lifecycle. Done means snapshots compact retained state, limit replay to newer changes, and improve recovery without losing correctness.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
database, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.