HarperFast / HarperFast/harper

Bulk vector read/scan API for HNSW-indexed attributes

Open
#1,774 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Add a supported **bulk read/scan API for the vectors stored under an HNSW-indexed attribute**, so a workload can iterate an index's vectors without decoding every full record for one field.

## Motivation

Several AI-data workloads need to read stored embeddings back out in bulk — not to search, but to *process* them:

- **Clustering** (k-means/GMM over a corpus — e.g. the RAPTOR summary-tree build loop in #1244 Thread B)
- **Re-embedding / migration** when a model changes
- **Export / offline analysis / dimensionality checks**

Today there's no scan surface. The only path is iterating the primary store (`primaryStore.getRange`) and reading `record[embeddingAttr]`, which **decodes the whole record to pull one array**. At ~1M×768-dim that's a lot of wasted decode. The HNSW index store *does* hold the vectors and is rangeable, but there's no supported API to read them back, and the entries are int8-quantized by default.

## The int8 question (measured — it's fine)

The int8 nodes store a per-vector `scale`, so the vectors reconstruct as `bin × scale`. I measured whether that precision is good enough to cluster on (harper#1244):

- Mean relative L2 reconstruction error ≈ **1.0%**.
- k-means on reconstructed-int8 vs float32 vectors: **98.6% agreement** (permutation-invariant Rand index); cluster→label purity identical.
- For calibration, two float32 runs from *different seeds* agree at only ~30% — i.e. **k-means is far more sensitive to its own init than to int8 reconstruction error.**

So a scan API can cheaply hand back **reconstructed int8 vectors** for quantized indexes (and full-precision for `quantization: "none"`) without needing a separate full-precision store.

## Sketch

A read-only iterator over an HNSW-indexed attribute's vectors, keyed by primaryKey, e.g.:

```javascript
for await (const { id, vector } of Table.scanVectors('embedding', { /* precision: 'stored' | 'exact' */ })) { … }
```

- `precision: 'stored'` (default) → reconstructed int8 (fast, ~1% error, adequate for clustering).
- `precision: 'exact'` → full-precision (falls back to reading the record's attribute where the index is quantized).
- Streams/batches; snapshot-consistent with the read txn.

## Notes

- This is the read-side complement to a broader vector-subsystem surface (#1235 lists a models/vector audit; #1277 covers query-time embedding).
- Filed out of the #1244 friction list (the "bulk vector export for clustering — is iterating records sufficient at 1M rows?" open question the issue itself raised — answer: a scan API is worth having).

🤖 Filed by Claude (Opus 4.8) on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the current primaryStore.getRange path and the HNSW index store's range access to understand how vectors and quantization metadata are stored. Define the supported scan surface around the proposed Table.scanVectors entry point, including stored and exact precision, streaming or batches, and read-transaction snapshot consistency. Done means callers can iterate vectors by primaryKey without decoding full records for the stored-precision path.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.