apache / apache/incubator-resilientdb

Add ResTier: Tiered Storage Engine with IPFS Cold Tier for ResilientDB

Open
#261 0 comments 0 reactions 0 assignees Claimed by @joshirajaram View on GitHub
Dominant language
C++
Stars
169
Forks
277
Avg merge
18h 51m
Merged PRs (30d)
1

Description

### Summary

ResilientDB currently persists all data in node-local LevelDB or MemoryDB storage indefinitely. Since ResilientDB is an append-only ledger by design, storage grows unboundedly in long-running production environments. There is no mechanism to offload historical or archival data, even though such data is accessed far less frequently than recent state. This leads to SSD bloat and makes the system impractical for large-scale deployments.
This issue proposes introducing ResTier, a tiered storage engine that transparently migrates cold data (calculated based on timestamp and configured threshold) from hot storage (MemoryDB/LevelDB) to IPFS as a decentralized, content-addressed cold tier - without any overhead on the write path or modifications to the PBFT consensus process.

### Motivation

The current single-tier storage model restrictions:

Unbounded local SSD growth leading to vertical scaling of nodes
Cost-effective archival of historical ledger data - historical data is less frequently queried
Separation of hot (frequently accessed) and cold (archival) data

Applications running ResilientDB at scale must manage storage externally. ResTier solves this natively within the storage layer.

### Proposed Changes - PR link: https://github.com/apache/incubator-resilientdb/pull/260

_IPFS Client_ - Introduce an HTTP-based IPFS client against the Kubo RPC API
Support Add, Cat, Exists, Unpin operations with retry logic
Add IPFSConfig proto for endpoint, timeout, and retry configuration

_Tiered Storage_ - Introduce TieredStorage as a three-tier orchestrator (HOT → WARM → COLD)
Background migration thread moves cold-eligible keys to IPFS based on configurable checkpoint thresholds
Transparent read fallback across tiers with no changes to existing KV interfaces
Manifest persistence to warm LevelDB for crash recovery with zero data loss

_Secondary Index Manifest_ - Add SecondaryIndex virtual interface for key-to-CID lookups
Implement InMemoryHashIndex for O(1) cold read resolution
Persist index manifest to warm LevelDB across restarts

_Storage Backend Changes_ - Add DeletableStorage interface with DeleteKey to support hot tier eviction
Extend LevelDB and MemoryDB with DeleteKey, mutex-protected batch writes, and timer instrumentation

_KV Service & Configuration_ - Refactor backend selection via StorageConfig enum supporting 4 storage modes: MEMORYDB, LEVELDB, TIERED (LevelDB->IPFS), TIERED (MemoryDB->IPFS)
Add server_tiered.config, server_tiered_leveldb.config, server_tiered_memorydb.config

_Migration Daemon_ - Add standalone sidecar process for orchestrating cold data migration independently of the main service

### Compatibility

Fully optional module - existing KV applications remain unaffected
No modifications to consensus or ledger protocols
Tiered mode enabled via runtime configuration only
Each PBFT replica migrates independently with no cross-node coordination

### Validation

Unit tests with stub IPFS client covering migration logic and read fallback
IPFS integration test against a live Kubo node
Crash recovery correctness verified across all four migration windows

### Expected Benefits

Handles unbounded SSD disk growth for long-running ResilientDB deployments
Enables cost-effective archival of historical ledger data via IPFS
Provides O(1) cold read performance via in-memory secondary index
Zero write-path overhead - PBFT consensus latency is fully preserved
Built-in deduplication and trustless content verification via IPFS CIDs

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no implementation file paths; begin with PR #260 and its proposed IPFS client, TieredStorage, manifest/index, and configuration entry points. Use the listed unit, live-IPFS integration, and crash-recovery validations to establish completion, noting that the linked PR means this work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.