Implementation of Spooky Compaction in Pebble

In this [paper](https://vldb.org/pvldb/vol15/p3071-dayan.pdf), there's description of a somewhat novel compaction algorithm (Spooky), that makes changes to how sst files are aligned with the lowest (largest) level to incur a much smaller Write Amplification while keeping Space and Read Amplification lower.

We're in the midst of developing this internally for usage, but I was wondering if this is something you would be open to committing to the OSS repo. There are 2 different algorithms within Spooky, 2L-Spooky and Lazy Leveled Spooky that seem to have the largest gains depending on the workload.

Example of the gains found in testing:

## Workload A — Update-heavy (fixed 5M keyspace, ~515 MiB live)

| Metric | Leveled | 2L-Spooky | Lazy | Tiered |
|---------------------------|------------|----------------|------------|------------|
| **Write-amp** (table) | 5.519 | **4.447** | 4.848 | 5.174 |
| **Read-amp** | 4 | **2** | 7 | 7 |
| **Space-amp** (disk/live) | 1.094 | **0.995** | 1.033 | 1.854 |
| Disk usage | 563.4 MiB | **512.2 MiB** | 532.0 MiB | 954.9 MiB |
| Table bytes written | 10.85 GiB | **8.75 GiB** | 9.53 GiB | 10.17 GiB |
| CPU time (user+sys) | 1m21.3s | **1m06.3s** | 1m10.7s | 1m14.9s |
| Wall (load+quiesce) | 1m26.0s | **1m00.3s** | 1m08.4s | 1m14.1s |
| Cumulative heap alloc | 10.05 GiB | **9.72 GiB** | 9.92 GiB | 9.93 GiB |
| Mallocs | 27,441,972 | **27,194,740** | 27,264,106 | 27,273,095 |
| GC cycles | 4,750 | **4,150** | 4,856 | 4,931 |
| **Total compactions** | 878 | **52** | 86 | 88 |
| — default | 276 | 9 | 43 | 44 |
| — spooky-preemptive | 0 | 16 | 25 | 24 |
| — spooky-tiering | 0 | 0 | 17 | 20 |
| — move | 602 | 27 | 1 | 0 |

**Winner: 2L-Spooky** — best on write-amp, read-amp, space-amp, CPU, allocations,
and compaction count simultaneously. The 878→52 compaction-count collapse is the
mechanism behind the rest.

## Workload B — Insert-heavy (keyspace = ops = 20M, ~2.01 GiB live)

| Metric | Leveled | 2L-Spooky | Lazy | Tiered |
|---------------------------|------------|----------------|--------------|---------------|
| **Write-amp** (table) | 6.749 | 8.074 | **5.984** | 5.936† |
| **Read-amp** | 5 | 3 | **3** | 9 |
| **Space-amp** (disk/live) | 0.676 | 0.650 | **0.624** | 0.773 |
| Disk usage | 1.36 GiB | 1.31 GiB | **1.25 GiB** | 1.56 GiB |
| Table bytes written | 13.76 GiB | 16.46 GiB | 12.20 GiB | **12.10 GiB** |
| CPU time (user+sys) | 1m29.8s | 1m45.6s | 1m23.8s | **1m22.8s** |
| Wall (load+quiesce) | 1m45.0s | 2m05.2s | 1m28.1s | **1m27.1s** |
| Cumulative heap alloc | 10.18 GiB | **9.83 GiB** | 9.96 GiB | 9.96 GiB |
| Mallocs | 27,492,752 | **27,295,109** | 27,293,748 | 27,293,199 |
| GC cycles | 4,761 | **3,929** | 4,711 | 4,681 |
| **Total compactions** | 1042 | 148 | 81 | **79** |
| — default | 293 | 54 | 43 | 41 |
| — spooky-preemptive | 0 | 14 | 20 | 21 |
| — spooky-tiering | 0 | 0 | 17 | 17 |
| — move | 749 | 80 | 1 | 0 |

† Tiered's 5.936 is statistically tied with Lazy's 5.984 (~0.8%, within noise).

**Winner: Lazy-leveling** — the only Spooky variant to beat plain Leveled on
write-amp on *any* workload, while also cutting read-amp 5→3 and posting the
lowest space-amp. 2L-Spooky is the *worst* here (8.074): it repeatedly folds
into a bottom level that is still growing.

We're going to be rolling this out in our fork of pebble and we want to offer as part of the upstream if the maintainers are willing to support it. Additonally, we've also added the following features to our fork:

1. SharedCache persists across restarts
2. SharedCache can be used concurrently by multiple pebble's
3. Adding a compaction filter for easy deletion of data without the bloat of tombstones

If there's any interest in these features, we're quite happy to file PRs and upstream.

Jira issue: PEBBLE-1454

Start with the linked Spooky paper and compare its 2L-Spooky and Lazy Leveled Spooky algorithms with Pebble's existing compaction behavior. Clarify with maintainers which algorithm and which of the additional SharedCache or compaction-filter features are in scope. Validate any implementation against the update-heavy and insert-heavy workloads described in the issue, with the reported amplification and compaction metrics as completion criteria.

Written by the indexing model from the issue text.