Integrating Rafture Information Dispersal with Filecoin

# Open Grant Proposal: Integrating Rafture Coding with FileCoin

**Project Name:** Integrating Rafture coding with FileCoin

**Proposal Category:** `Storage`

**Individual or Entity Name:** fftech lab

**Proposer:** `Rithwikkerur`

**Project Repo(s):** Please list Github repos used for this project work.

We will use this erasure coding library: https://github.com/AndersTrier/reed-solomon-simd

We will create a new repo for Filecoin during Milestone 1

**(Optional) Filecoin ecosystem affiliations:** None

**(Optional) Technical Sponsor:** None

**Do you agree to open source all work you do on behalf of this RFP under the MIT/Apache-2 dual-license?:**
**Yes**

---

# Project Summary

The main objective of this proposal is to reduce storage costs in FileCoin, while at the same time simplifying the core storage layer. The approach we explore is built on two novel tenets:

- A unique strategy for file encoding and dissemination of encoded pieces
- A method for post-dissemination pruning of file storage

In order to target the ambitious goal above — cutting down storage to a half — we propose to integrate Rafture, a new method for asynchronous information dissemination (under submission).
To understand the novel idea behind Rafture, we first underscore the main challenge distributed systems face.

Distributed protocols must be designed to complete when messages are received from N-F nodes, preparing for the worst- but rare- case of F failures (including the catastrophic, Byzantine kind). In steady state, the remaining F nodes may be healthy, but their resources are not counted. This results in over-provisioning of storage for encoded data. Typically Byzantine protocols use an (F+1, N) encoding scheme to generate N fragments such that any F+1 can be used to reconstruct the data item. The sender then sends a single fragment to each node and waits to receive responses. In order to commit, the sender needs to receive responses from at least 2F+1 nodes to guarantee that there are at least F+1 honest nodes to reconstruct the data item if needed.

Rafture addresses this conundrum with a two-stage approach:

**First**, the core protocol estimates the actual number ("Fa") of unresponsive nodes, rather than the worst case F. Rafture then uses a (2F+1, (2F+1)*N) encoding scheme to generate a very large number of unique fragments. Rafture can then overprovision and include extra fragments in the initial dissemination to nodes. Instead of sending a single fragment to each node, Rafture can send up to 2F+1 unique fragments to each node based on its estimation of network health. Completion is achieved quickly if messages are received from N-Fa nodes, or more slowly if fewer nodes respond. This flexibility introduces a natural tradeoff between storage efficiency and latency.

**Second**, after the initial dissemination is completed, Rafture continues monitoring for responses. When additional responses are received from up to N nodes, the system prunes the information they store accordingly. If a sender receives N responses to its initial dissemination attempt it can instruct nodes to drop all but 1 shard. Since we use an encoding where k=2F+1 instead of the classic k=F+1, if the sender receives responses from all nodes, storage can be reduced by a factor of 2. A key technique enabling seamless pruning is an agile encoding scheme that allows nodes to discard encoded pieces on-the-fly. Crucially, this is done without maintaining complex meta-information. Furthermore, adjustments can be done by individual nodes autonomously, without global coordination.

---

## Impact

At its core, Rafture is an information dispersal solution consisting of two innovative ingredients:

1. An encoding strategy that enables flexible information dissemination regimes under different network conditions
2. A method for post-dissemination pruning, allowing systems to adapt storage costs after dissemination completes

Importantly, both components are simple and practical to implement. Rafture employs a fixed-threshold erasure code while varying the number of encoded pieces. Recovery is always possible using the same interpolation method, without any complex bookkeeping or metadata maintenance. Furthermore, post-dissemination pruning is carried out autonomously by nodes without global coordination, simply by each node observing network conditions locally.

In addition to concrete benefits to the FileCoin technology, our exploration aims to advance the state-of-art of decentralized storage systems. Spreading and storing erasure-coded data in distributed systems effectively is challenging in real settings. Practical deployments, like FileCoin, must contend with unpredictable network latencies. Our exploration aims to establish new foundations that focus on post-dissemination optimization and addresses both theory and practice.

---

## Outcomes

We have integrated our Information Dispersal Approach into a Crash Fault Tolerant system like Raft. In this implementation we have already demonstrated quantifiable storage savings and latency improvements, particularly under degraded network conditions. Building upon this foundation, we want to deliver similar results when adapting our IDA to work for Byzantine cases in FileCoin. We want to:

- Integrate our IDA approach into FileCoin either by modifying the current encoding or by utilizing a completely new coding scheme
- Create a benchmark suite to demonstrate the storage savings and latency savings from a client's perspective
- Produce an academic paper detailing our methodology and evaluation results

---

## Data Onboarding

We estimate at most 100-200GB to develop and test the full erasure coding pipeline.

---

## Adoption, Reach, and Growth Strategies

We are proposing an infrastructure-level change that would reduce the long-term storage costs of data on FileCoin. By cutting storage costs in half, we make it cheaper for the user to store data on the FileCoin network and reduce the storage overhead for data providers. With over 3,500 active storage providers on the FileCoin network, this improvement has the potential for wide adoption, and we intend to propose it as a Filecoin Improvement Proposal (FIP) upon successful validation.

---

## Development Roadmap

This development roadmap is divided into four distinct milestones. It transitions the FileCoin network from its current approach to the Rafture system.

### Milestone 1: Dynamic Failure Estimation (Fa) & Protocol Adaptation

**Timeframe:** September 2026 – October 2026 | **Funding:** $10,000

The first phase focuses on altering the core assumptions of the FileCoin dissemination protocol. Instead of hardcoding operations to the worst-case failure scenario (F), the system will dynamically estimate actual network health.

**Expected Functionality:**
- Implementation of a network monitoring module that accurately estimates Fa (the actual number of faulty nodes) in real-time
- Develop an understanding of the responsiveness of the network and what coding parameters would be most optimal

**Funding Rationale:** Estimated allocation for 2 months of solo engineering & testing infrastructure

---

### Milestone 2: Legacy Pruning Validation & Core Rafture Integration

**Timeframe:** November 2026 – December 2026 | **Funding:** $10,000

Phase two tackles the novel tenets of the proposal directly — proving the viability of post-dissemination pruning before replacing the legacy core with Rafture's fixed-threshold encoding.

**Expected Functionality:**
- Validation of post-dissemination pruning using the legacy encoding scheme as a proof-of-concept for safe data discarding within the FileCoin architecture
- Complete deprecation of the legacy encoding layer, replacing the core storage mechanism with the Rafture encoding scheme
- Establishment of the new baseline: information dispersed using a fixed-threshold erasure code with a variable number of encoded pieces, allowing for standard interpolation recovery without complex metadata or bookkeeping

**Funding Rationale:** Estimated allocation for 2 months of storage engineering

---

### Milestone 3: Autonomous Post-Dissemination Pruning

**Timeframe:** January 2027 – February 2027 | **Funding:** $10,000

With the Rafture encoding layer in place, Milestone 3 focuses on achieving the primary objective: cutting storage requirements by up to a factor of 2 through localized, decentralized pruning.

**Expected Functionality:**
- Deployment of the post-dissemination piece-availability monitor on individual FileCoin storage nodes
- Nodes can autonomously and safely discard redundant encoded pieces on-the-fly as late responses are received from up to N healthy nodes
- The pruning process functions entirely locally, achieving maximum storage efficiency without requiring any global coordination or introducing new networking overhead to FileCoin's consensus

**Funding Rationale:** Estimated allocation for 2 months of node-level daemon development and optimization

---

### Milestone 4: Testing

**Timeframe:** March 2027 – April 2027 | **Funding:** $10,000

The final phase polishes the newly upgraded FileCoin network. With storage requirements slashed, the focus shifts to testing and benchmarking

**Expected Functionality:**
- Deploy and test our approach on testnet to verify the 2x storage savings
- Produce a benchmark report documenting storage and latency measurements under various network conditions
- Publish simulation results comparing Rafture encoding against the current FileCoin baseline

**Funding Rationale:** Estimated allocation for comprehensive benchmarking and testing

---

## Total Budget Requested

| Milestone # | Description | Deliverables | Completion Date | Funding |
|---|---|---|---|---|
| 1 | Dynamic Failure Estimation & Protocol Adaptation | Network monitoring module, protocol modifications, testnet demo | October 2026 | $10,000 |
| 2 | Legacy Pruning Validation & Core Rafture Integration | Pruning validation, Rafture encoding layer, new baseline | December 2026 | $10,000 |
| 3 | Autonomous Post-Dissemination Pruning | Pruning daemon, autonomous node optimization | February 2027 | $10,000 |
| 4 | Testing | benchmarking and testing report | April 2027 | $10,000 |
| **Total** | | | | **$40,000** |

---

## Maintenance and Upgrade Plans

There are new novel erasure coding techniques that are starting to surface that make reconstruction more optimal. An upgrade plan would be to integrate these new coding libraries to make reconstruction more optimal.

---

# Team

## Team Members

- Rithwik Kerur
- Dahlia Malkhi

## Team Member LinkedIn Profiles

https://www.linkedin.com/in/rithwikkerur/
https://www.linkedin.com/in/dahlia-malkhi-8b836a5/

## Team Website

https://fiftech.cs.ucsb.edu/

## Relevant Experience

We have expertise in both distributed systems and erasure coding. We currently have a paper under submission for integrating Rafture encoding with Raft. We are also exploring integrating Rafture encoding with other Byzantine consensus and decentralized storage systems.

## Team Code Repositories

https://github.com/RithwikKerur/FlexRaft-Code

---

# Additional Information

Rafture Paper: https://arxiv.org/abs/2603.24761

Contact: rkerur@ucsb.edu