ruvnet / ruvnet/RuVector

RVF compaction: policy is never consulted, and the WASM backend reports constants where it has real data

Open
#753 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
4.5k
Forks
603
Avg merge
23h 32m
Merged PRs (30d)
59

Description

Three related defects around RVF compaction, found while qualifying RVF as an evidence store for a downstream QE system. All verified against main @ 597be6a75. Each is independently fixable; two are one-line-ish, one is a design decision.

Happy to send PRs for all three — see the split at the bottom.


1. The compaction scheduling policy is dead code

crates/rvf/rvf-runtime/src/compaction.rs implements the policy from spec 10 §7 in full, with passing unit tests:

  • CompactionThresholdsdead_space_ratio 0.20, max_segment_count 32, min_interval_secs 60, emergency_ratio 0.70
  • evaluate_triggers()CompactionDecision::{None,Normal,Emergency}
  • select_segments() — tombstoned first, then small VEC_SEGs, then by age

Every one is marked #[allow(dead_code)], and the only callers are the module's own #[cfg(test)] tests:

$ grep -rn "evaluate_triggers" --include='*.rs' . | grep -v /target/
crates/rvf/rvf-runtime/src/compaction.rs:53:pub(crate) fn evaluate_triggers(    <- definition
crates/rvf/rvf-runtime/src/compaction.rs:165,171,177,183,189                   <- tests only

select_segments is the same — definition plus one test. So RvfStore::compact() compacts whenever it is called, and the documented triggers never gate anything. rvf-cli's compact.rs calls store.compact() directly with no evaluation either.

Why it matters downstream: consumers have no signal for when to compact, so they call it unconditionally on a schedule. Compacting an append-only store far more often than the policy intends is at minimum wasted IO.

Two honest options, and the choice is yours:

  • wire evaluate_triggers into a compact_if_needed() (or into compact() behind a flag), or
  • if compaction is deliberately caller-scheduled, delete the dead policy and say so — an implemented-and-tested policy that nothing consults reads as enforced when it is not.

2. WasmBackend.compact() reports success without doing anything

npm/packages/rvf/src/backend.ts:863

async compact(): Promise<RvfCompactionResult> {
  return { segmentsCompacted: 0, bytesReclaimed: 0, epoch: 0 };
}

No handle check, no WASM call. And there is no compaction export to call — the C-ABI block in crates/rvf/rvf-wasm/src/lib.rs runs rvf_store_creatervf_store_close with no rvf_store_compact under any name, and npm/packages/rvf-wasm/pkg/rvf_wasm.d.ts confirms it.

The same class throws for every other unsupported operation — deleteByFilter at :859, then fileId, parentId, lineageDepth, derive, branch, freeze, embedKernel, extractKernel, embedEbpf, extractEbpf, segments. Eleven precedents for RvfError(BackendNotFound, '... not supported in WASM backend'); compact() is the one that fakes success instead.

A caller cannot distinguish "compaction ran and there was nothing to reclaim" from "compaction is not implemented here". The WASM store does retain soft-deleted entries (export() filters them), so there is genuinely something a caller might expect to be reclaimed.

3. WasmBackend.status() returns deadSpaceRatio: 0 while the buffer carries the real number

crates/rvf/rvf-wasm/src/store.rs:142 writes a documented 20-byte status buffer:

offset field
0 live entries
4 dimension
8 metric
12 total entries
16 deleted entries

npm/packages/rvf/src/backend.ts:867 allocates all 20 bytes, reads offsets 0 and 4, frees the buffer, and hardcodes the rest:

return {
  totalVectors,
  totalSegments: 1,
  fileSizeBytes: 0,
  epoch: 0,
  profileId: 0,
  compactionState: 'idle',
  deadSpaceRatio: 0,   // <- offsets 12 and 16 were just discarded
  readOnly: false,
};

deleted / total is the dead-space ratio, and both were in the buffer.

Reproduction

Driving WasmBackend with a fake that writes the buffer exactly as store.rs does:

wasm buffer said:  total=10 deleted=7   => real dead space 0.70
status() returned: deadSpaceRatio = 0
a consumer gating on the documented policy (dead > 0.20 || segs > 32):
  would compact? false      <- never fires, at the 0.70 EMERGENCY threshold

So any consumer that correctly gates on the documented policy is permanently inert on the WASM backend, and the failure is silent — the field is present and plausible, just constant.

Scoping this one narrowly, on purpose: only deadSpaceRatio is recoverable from that buffer. Offset 12 is total vector entries, not segments, and the buffer carries no file size or epoch. totalSegments: 1 may be a deliberate abstraction for an in-memory store — if so it is worth a comment, but it is not part of this bug.

NodeBackend is unaffected: it maps dead_space_ratio from the native status (backend.ts:1180).


Suggested PR split

  • PR A — #2 + #3. Both are in npm/packages/rvf, self-contained, no behaviour change for NodeBackend: compute deadSpaceRatio from offsets 12/16, and make compact() throw BackendNotFound like its eleven siblings.
  • PR B — #1. Separate, because wiring the trigger policy changes compaction behaviour for every consumer and deserves its own review.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/rvf/rvf-runtime/src/compaction.rs and its unit tests, then inspect npm/packages/rvf/src/backend.ts alongside crates/rvf/rvf-wasm/src/store.rs and rvf-wasm.d.ts. Reproduce the status-buffer values and review the existing unsupported-operation errors. Done means the WASM status exposes its stored dead-space ratio, unsupported compaction is reported consistently, and the policy integration is handled as a separate reviewed change.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, typescript, wasm
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.