apple / apple/foundationdb

Incorporate upstream improvements to SQLite B-Tree code

Open
#2,239 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The `ssd` storage engine uses SQLite's btree code, imported many years ago, for its property of excellent testability. As many years have passed, further development has occurred upstream, which we've never included back into FDB. To further complicate the issue, we've done our own improvements to the SQLite code, to add checksumming and improve the cleanup behaviors of the code.

I've done a comparison of our btree.c vs the most recent release, and it looks like there were two major changes to the code. The first looks like something to allow the page cache to be shared between multiple processes, which would bring us no benefit to include. The second is detecting and handling corrupted SQLite databases better. This is superior to our own modifications here, because simulation only corrupts things we modify, and there are some changes to handle some of the fixed contents that we never change (like the page size recorded in the database).

It's possible that there's other useful changes, as I didn't compare any files outside of btree.c

Contributor guide

Open the contributing guide

Research direction

Start by comparing the SSD storage engine's btree.c with the most recent upstream SQLite version, focusing on the corruption-detection changes described in the issue. Review the local checksumming and cleanup modifications before deciding what can be integrated, then check whether other SQLite files contain relevant changes. Done means useful upstream improvements are incorporated without losing FoundationDB-specific behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sqlite
Domain
databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.