celestiaorg / celestiaorg/go-header

store: Synchronous API

Open
#244 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
21
Forks
33
Avg merge
5d 20h
Merged PRs (30d)
7

Description

Current writes and read in the Store are asynchronous. Writes via `Append` cannot be accessed via `Get` immediately after write(same as for `Head` after #239).

However, doing this with our requirements is not trivial. We need to keep writes IO free and return as early as possible as its currently done. With introduction of contiguous head in #239 this becomes even more problematic, as each write has to advance this head. Advancing, in its current implementation, always hits the disk by checking if the next disk is available locally and setting this new header as new contiguous head. Allowing each write to hit the disk is gonna be a major performance regression, so the solution to that must be found.

Conceptually, the solution is possible by tracking the gaps after the contiguous head within the Store. As Store has full control over what gets written into it, it can retain header ranges that didn't contribute to a contiguous chain and produces gaps. Then, whenever any new header is written, the Store is gonna check tracked header and see if it advance the contiguous head without hitting the disk. This should work well in conjuction with autoadvancing on `Start` implemented as #239, allowing writes to avoid disk hits in case tracked header gaps are cleared out after the restart.

Contributor guide

Open the contributing guide

Research direction

Start by reading the Store implementation and the Append, Get, Head, and Start paths, then review the contiguous-head work in #239. Trace how writes advance the contiguous head and identify where disk access occurs. Done means writes remain IO-free and return early while Get and Head can observe them immediately without a per-write disk hit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.