apple / apple/foundationdb

Create knob to only perform SQLite vacuuming above a given file size.

Open
#1,578 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

SQLite vacuuming, which is the relocating of used pages at the end of the database file into free pages within the file to enable file shrinking, has a very high IO cost due to the updating of parent and metadata pages spread throughout the database file.

In a deployment scenario where disk space is being set aside for exclusive use by the storage server instance, this file shrinkage is usually unnecessary and a waste of IO and CPU so it could be disabled.

However, in the event that the new page write rate temporarily outpaces the lazy deletion process (which gradually clears subtrees and frees their pages) the freelist may be exhausted and the database file will grow as the new pages are added to the end of the file, even though the total KV bytes size (after clears are processed) may not necessarily be growing. This can cause the database file to grow larger than the space set aside for it so SQLite vacuuming should be performed to shrink the file.

To enable this, I think there should be a knob, perhaps SPRING_CLEANING_VACUUM_MIN_FILE_SIZE, which will set a file size limit over which SQLite vacuuming is turned on. Its default should probably be 0.

Possibly a second knob could be defined as well which sets a lower limit on the amount of free space on the filesystem before vacuuming is turned on.

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is named. Start by locating the SQLite vacuuming path and the lazy deletion and freelist handling described in the issue. Define the file-size threshold behavior and its default, then verify vacuuming remains off below the threshold and activates above it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sqlite
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.