RVF compaction: policy is never consulted, and the WASM backend reports constants where it has real data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 4.5k
- Forks
- 603
- Avg merge
- 23h 32m
- Merged PRs (30d)
- 59
Description
Three related defects around RVF compaction, found while qualifying RVF as an evidence store for a downstream QE system. All verified against main @ 597be6a75. Each is independently fixable; two are one-line-ish, one is a design decision.
Happy to send PRs for all three — see the split at the bottom.
1. The compaction scheduling policy is dead code
crates/rvf/rvf-runtime/src/compaction.rs implements the policy from spec 10 §7 in full, with passing unit tests:
CompactionThresholds—dead_space_ratio 0.20,max_segment_count 32,min_interval_secs 60,emergency_ratio 0.70evaluate_triggers()→CompactionDecision::{None,Normal,Emergency}select_segments()— tombstoned first, then small VEC_SEGs, then by age
Every one is marked #[allow(dead_code)], and the only callers are the module's own #[cfg(test)] tests:
$ grep -rn "evaluate_triggers" --include='*.rs' . | grep -v /target/
crates/rvf/rvf-runtime/src/compaction.rs:53:pub(crate) fn evaluate_triggers( <- definition
crates/rvf/rvf-runtime/src/compaction.rs:165,171,177,183,189 <- tests only
select_segments is the same — definition plus one test. So RvfStore::compact() compacts whenever it is called, and the documented triggers never gate anything. rvf-cli's compact.rs calls store.compact() directly with no evaluation either.
Why it matters downstream: consumers have no signal for when to compact, so they call it unconditionally on a schedule. Compacting an append-only store far more often than the policy intends is at minimum wasted IO.
Two honest options, and the choice is yours:
- wire
evaluate_triggersinto acompact_if_needed()(or intocompact()behind a flag), or - if compaction is deliberately caller-scheduled, delete the dead policy and say so — an implemented-and-tested policy that nothing consults reads as enforced when it is not.
2. WasmBackend.compact() reports success without doing anything
npm/packages/rvf/src/backend.ts:863
async compact(): Promise<RvfCompactionResult> {
return { segmentsCompacted: 0, bytesReclaimed: 0, epoch: 0 };
}
No handle check, no WASM call. And there is no compaction export to call — the C-ABI block in crates/rvf/rvf-wasm/src/lib.rs runs rvf_store_create … rvf_store_close with no rvf_store_compact under any name, and npm/packages/rvf-wasm/pkg/rvf_wasm.d.ts confirms it.
The same class throws for every other unsupported operation — deleteByFilter at :859, then fileId, parentId, lineageDepth, derive, branch, freeze, embedKernel, extractKernel, embedEbpf, extractEbpf, segments. Eleven precedents for RvfError(BackendNotFound, '... not supported in WASM backend'); compact() is the one that fakes success instead.
A caller cannot distinguish "compaction ran and there was nothing to reclaim" from "compaction is not implemented here". The WASM store does retain soft-deleted entries (export() filters them), so there is genuinely something a caller might expect to be reclaimed.
3. WasmBackend.status() returns deadSpaceRatio: 0 while the buffer carries the real number
crates/rvf/rvf-wasm/src/store.rs:142 writes a documented 20-byte status buffer:
| offset | field |
|---|---|
| 0 | live entries |
| 4 | dimension |
| 8 | metric |
| 12 | total entries |
| 16 | deleted entries |
npm/packages/rvf/src/backend.ts:867 allocates all 20 bytes, reads offsets 0 and 4, frees the buffer, and hardcodes the rest:
return {
totalVectors,
totalSegments: 1,
fileSizeBytes: 0,
epoch: 0,
profileId: 0,
compactionState: 'idle',
deadSpaceRatio: 0, // <- offsets 12 and 16 were just discarded
readOnly: false,
};
deleted / total is the dead-space ratio, and both were in the buffer.
Reproduction
Driving WasmBackend with a fake that writes the buffer exactly as store.rs does:
wasm buffer said: total=10 deleted=7 => real dead space 0.70
status() returned: deadSpaceRatio = 0
a consumer gating on the documented policy (dead > 0.20 || segs > 32):
would compact? false <- never fires, at the 0.70 EMERGENCY threshold
So any consumer that correctly gates on the documented policy is permanently inert on the WASM backend, and the failure is silent — the field is present and plausible, just constant.
Scoping this one narrowly, on purpose: only deadSpaceRatio is recoverable from that buffer. Offset 12 is total vector entries, not segments, and the buffer carries no file size or epoch. totalSegments: 1 may be a deliberate abstraction for an in-memory store — if so it is worth a comment, but it is not part of this bug.
NodeBackend is unaffected: it maps dead_space_ratio from the native status (backend.ts:1180).
Suggested PR split
- PR A — #2 + #3. Both are in
npm/packages/rvf, self-contained, no behaviour change forNodeBackend: computedeadSpaceRatiofrom offsets 12/16, and makecompact()throwBackendNotFoundlike its eleven siblings. - PR B — #1. Separate, because wiring the trigger policy changes compaction behaviour for every consumer and deserves its own review.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crates/rvf/rvf-runtime/src/compaction.rs and its unit tests, then inspect npm/packages/rvf/src/backend.ts alongside crates/rvf/rvf-wasm/src/store.rs and rvf-wasm.d.ts. Reproduce the status-buffer values and review the existing unsupported-operation errors. Done means the WASM status exposes its stored dead-space ratio, unsupported compaction is reported consistently, and the policy integration is handled as a separate reviewed change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, typescript, wasm
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100