apache / apache/incubator-resilientdb
Add ResTier: Tiered Storage Engine with IPFS Cold Tier for ResilientDB
- Dominant language
- C++
- Stars
- 169
- Forks
- 277
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 1
Description
### Summary
ResilientDB currently persists all data in node-local LevelDB or MemoryDB storage indefinitely. Since ResilientDB is an append-only ledger by design, storage grows unboundedly in long-running production environments. There is no mechanism to offload historical or archival data, even though such data is accessed far less frequently than recent state. This leads to SSD bloat and makes the system impractical for large-scale deployments.
This issue proposes introducing ResTier, a tiered storage engine that transparently migrates cold data (calculated based on timestamp and configured threshold) from hot storage (MemoryDB/LevelDB) to IPFS as a decentralized, content-addressed cold tier - without any overhead on the write path or modifications to the PBFT consensus process.
### Motivation
The current single-tier storage model restrictions:
Unbounded local SSD growth leading to vertical scaling of nodes
Cost-effective archival of historical ledger data - historical data is less frequently queried
Separation of hot (frequently accessed) and cold (archival) data
Applications running ResilientDB at scale must manage storage externally. ResTier solves this natively within the storage layer.
### Proposed Changes - PR link: https://github.com/apache/incubator-resilientdb/pull/260
_IPFS Client_ - Introduce an HTTP-based IPFS client against the Kubo RPC API
Support Add, Cat, Exists, Unpin operations with retry logic
Add IPFSConfig proto for endpoint, timeout, and retry configuration
_Tiered Storage_ - Introduce TieredStorage as a three-tier orchestrator (HOT → WARM → COLD)
Background migration thread moves cold-eligible keys to IPFS based on configurable checkpoint thresholds
Transparent read fallback across tiers with no changes to existing KV interfaces
Manifest persistence to warm LevelDB for crash recovery with zero data loss
_Secondary Index Manifest_ - Add SecondaryIndex virtual interface for key-to-CID lookups
Implement InMemoryHashIndex for O(1) cold read resolution
Persist index manifest to warm LevelDB across restarts
_Storage Backend Changes_ - Add DeletableStorage interface with DeleteKey to support hot tier eviction
Extend LevelDB and MemoryDB with DeleteKey, mutex-protected batch writes, and timer instrumentation
_KV Service & Configuration_ - Refactor backend selection via StorageConfig enum supporting 4 storage modes: MEMORYDB, LEVELDB, TIERED (LevelDB->IPFS), TIERED (MemoryDB->IPFS)
Add server_tiered.config, server_tiered_leveldb.config, server_tiered_memorydb.config
_Migration Daemon_ - Add standalone sidecar process for orchestrating cold data migration independently of the main service
### Compatibility
Fully optional module - existing KV applications remain unaffected
No modifications to consensus or ledger protocols
Tiered mode enabled via runtime configuration only
Each PBFT replica migrates independently with no cross-node coordination
### Validation
Unit tests with stub IPFS client covering migration logic and read fallback
IPFS integration test against a live Kubo node
Crash recovery correctness verified across all four migration windows
### Expected Benefits
Handles unbounded SSD disk growth for long-running ResilientDB deployments
Enables cost-effective archival of historical ledger data via IPFS
Provides O(1) cold read performance via in-memory secondary index
Zero write-path overhead - PBFT consensus latency is fully preserved
Built-in deduplication and trustless content verification via IPFS CIDs
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no implementation file paths; begin with PR #260 and its proposed IPFS client, TieredStorage, manifest/index, and configuration entry points. Use the listed unit, live-IPFS integration, and crash-recovery validations to establish completion, noting that the linked PR means this work is already underway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100