alibaba / alibaba/neug

Support NULL vector properties: skip NULL vectors in HNSW index and distance functions

Open
#893 2 comments 0 reactions 1 assignee Claimed by @shirly121 View on GitHub
extension store
Dominant language
C++
Stars
164
Forks
33
Avg merge
1d 11h
Merged PRs (30d)
99

Description

# Support NULL vector properties (skip NULL vectors in HNSW index and distance functions)

Part of #885 (extending the null contract from in-memory column builders to storage-level property NULL semantics).

## Background

Vector properties currently have no NULL semantics in storage. A node created without its vector property materializes as an **all-zero vector**, because storage has no per-property validity — missing property = type default.

Consequences (documented as intended behavior in the `vector_search` extension docs, PR #851):

- **Ghost neighbors**: nodes that never had a vector are indexed as zero vectors and appear in ANN / brute-force similarity results at a fixed distance from the query (e.g. `vector_distance_l2(query, [0,0,0,0])`).
- **Zero vector is mathematically invalid for cosine**: cosine similarity with the zero vector is undefined; the index and distance functions currently accept it anyway.
- **No way to opt out**: `SET n.vec = NULL` is rejected with `ERR_INTERNAL_ERROR ("Property type mismatch")` — an unintuitive error for what is really "vector properties cannot be NULL".

## Proposal (pgvector-style NULL semantics)

1. **Storage**: persist a validity bitmap for vector (fixed-length ARRAY) property columns. The in-memory column layer already has validity support (`ValueColumn::valid_`, `push_back_null` — see #878/#882); extend it to storage, checkpoint, and WAL.
2. **DML**: missing vector property reads as NULL instead of the zero default; `SET n.vec = NULL` becomes legal and clears the vector.
3. **Index maintenance**: `HNSWVecSource` skips NULL rows (the index ID space covers valid vectors only). Null transitions update the HNSW index incrementally: setting a vector inserts into the index; setting NULL removes.
4. **Query**: `vector_distance_*(NULL, q)` returns NULL (row filtered from results); ANN index scans never return NULL-vector rows.

With this in place, the implicit zero-vector behavior disappears and the current documentation caveat can be removed.

## Checklist

- [ ] Persisted validity bitmap for vector property columns (checkpoint + WAL)
- [ ] `SET n.vec = NULL` support with a proper error message until then ("Vector property cannot be set to NULL")
- [ ] `HNSWVecSource` skips NULL rows; incremental add/remove on null transitions
- [ ] Distance functions return NULL for NULL vectors; ANN scans exclude them
- [ ] Regression tests: create-without-vector, SET NULL, null transition before/after checkpoint reopen, cosine/l2/ip metrics
- [ ] Update vector_search docs once NULL semantics land

## Relation to #885

#885 tracks the in-memory builder null contract. This issue is the storage-level counterpart for property NULL, vector columns first.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.