Incorporate upstream improvements to SQLite B-Tree code
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The `ssd` storage engine uses SQLite's btree code, imported many years ago, for its property of excellent testability. As many years have passed, further development has occurred upstream, which we've never included back into FDB. To further complicate the issue, we've done our own improvements to the SQLite code, to add checksumming and improve the cleanup behaviors of the code.
I've done a comparison of our btree.c vs the most recent release, and it looks like there were two major changes to the code. The first looks like something to allow the page cache to be shared between multiple processes, which would bring us no benefit to include. The second is detecting and handling corrupted SQLite databases better. This is superior to our own modifications here, because simulation only corrupts things we modify, and there are some changes to handle some of the fixed contents that we never change (like the page size recorded in the database).
It's possible that there's other useful changes, as I didn't compare any files outside of btree.c
Contributor guide
Research direction
Start by comparing the SSD storage engine's btree.c with the most recent upstream SQLite version, focusing on the corruption-detection changes described in the issue. Review the local checksumming and cleanup modifications before deciding what can be integrated, then check whether other SQLite files contain relevant changes. Done means useful upstream improvements are incorporated without losing FoundationDB-specific behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sqlite
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100