Support NULL vector properties: skip NULL vectors in HNSW index and distance functions
- Dominant language
- C++
- Stars
- 164
- Forks
- 33
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 99
Description
# Support NULL vector properties (skip NULL vectors in HNSW index and distance functions)
Part of #885 (extending the null contract from in-memory column builders to storage-level property NULL semantics).
## Background
Vector properties currently have no NULL semantics in storage. A node created without its vector property materializes as an **all-zero vector**, because storage has no per-property validity — missing property = type default.
Consequences (documented as intended behavior in the `vector_search` extension docs, PR #851):
- **Ghost neighbors**: nodes that never had a vector are indexed as zero vectors and appear in ANN / brute-force similarity results at a fixed distance from the query (e.g. `vector_distance_l2(query, [0,0,0,0])`).
- **Zero vector is mathematically invalid for cosine**: cosine similarity with the zero vector is undefined; the index and distance functions currently accept it anyway.
- **No way to opt out**: `SET n.vec = NULL` is rejected with `ERR_INTERNAL_ERROR ("Property type mismatch")` — an unintuitive error for what is really "vector properties cannot be NULL".
## Proposal (pgvector-style NULL semantics)
1. **Storage**: persist a validity bitmap for vector (fixed-length ARRAY) property columns. The in-memory column layer already has validity support (`ValueColumn::valid_`, `push_back_null` — see #878/#882); extend it to storage, checkpoint, and WAL.
2. **DML**: missing vector property reads as NULL instead of the zero default; `SET n.vec = NULL` becomes legal and clears the vector.
3. **Index maintenance**: `HNSWVecSource` skips NULL rows (the index ID space covers valid vectors only). Null transitions update the HNSW index incrementally: setting a vector inserts into the index; setting NULL removes.
4. **Query**: `vector_distance_*(NULL, q)` returns NULL (row filtered from results); ANN index scans never return NULL-vector rows.
With this in place, the implicit zero-vector behavior disappears and the current documentation caveat can be removed.
## Checklist
- [ ] Persisted validity bitmap for vector property columns (checkpoint + WAL)
- [ ] `SET n.vec = NULL` support with a proper error message until then ("Vector property cannot be set to NULL")
- [ ] `HNSWVecSource` skips NULL rows; incremental add/remove on null transitions
- [ ] Distance functions return NULL for NULL vectors; ANN scans exclude them
- [ ] Regression tests: create-without-vector, SET NULL, null transition before/after checkpoint reopen, cosine/l2/ip metrics
- [ ] Update vector_search docs once NULL semantics land
## Relation to #885
#885 tracks the in-memory builder null contract. This issue is the storage-level counterpart for property NULL, vector columns first.
Contributor guide
Assessment
This issue has not been assessed yet.