HarperFast / HarperFast/harper
search_by_conditions can't do vector/k-NN search on @embed / HNSW-indexed columns
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A table using the `@embed` GraphQL directive gets an auto-generated `[Float]` embedding column with an HNSW index (`resources/graphql.ts`, `resources/models/embedHook.ts`). But that vector column can only be queried via the REST/GraphQL Query `sort` clause (`sort: { attribute, target, distance }` → `Table.search()` → `comparator: 'sort'` → HNSW `customIndex.search`), never through the operations-API `search_by_conditions` call.
This is an API-parity gap: everything else on a table (equality, range, prefix conditions) is reachable through both the Query interface and `search_by_conditions`, but nearest-neighbor/similarity search is Query-only.
## Root cause
- `validation/searchValidator.ts:42-55` — `comparator` is a fixed Joi enum (`equals`, `contains`, `starts_with`, `ends_with`, `greater_than[_equal]`, `less_than[_equal]`, `between`, `not_equal`); there's no vector/similarity comparator, and every condition's `value` is required (scalar).
- `dataLayer/harperBridge/ResourceBridge.ts` `mapCondition` (~lines 62-75) normalizes each condition down to `{ attribute, comparator, value }`, so even if a caller attaches a `target` vector + `distance` to a condition, it's dropped before reaching the index.
- `resources/indexes/HierarchicalNavigableSmallWorld.ts` `search()` (~lines 925-963) only accepts `sort`/`lt`/`le` comparators and expects `target`/`distance` — which the `search_by_conditions` path never constructs.
- The MCP tool schema for search operations (`components/mcp/tools/schemas/operations.ts:148`) has the same narrower scalar-only comparator enum.
## Proposal
Extend `search_by_conditions` to accept a vector search condition — e.g. a `sort`-comparator condition (or a dedicated `nearest`/`similar_to` comparator) carrying `target` (query vector) and `distance` (`cosine`/`euclidean`/`dot`), validated against columns backed by an HNSW custom index, and routed through the same `Table.search()` / `customIndex.search()` path the Query `sort` clause already uses. This would let operations-API/MCP callers do k-NN search on `@embed` tables without going through GraphQL/REST.
## Current workaround
Vector search today must go through the Query/GraphQL/REST `sort` clause or the programmatic `search()` Resource API (which also supports `vectorFilter`, JS-only) — see `integrationTests/qa-scratch/vector-search.test.ts` for the supported pattern.
Contributor guide
Research direction
Read validation/searchValidator.ts and ResourceBridge.ts mapCondition first, then trace the existing sort path through Table.search() and resources/indexes/HierarchicalNavigableSmallWorld.ts. Check components/mcp/tools/schemas/operations.ts and integrationTests/qa-scratch/vector-search.test.ts. Done means operations-API and MCP callers can perform validated vector search on HNSW-backed columns with coverage for the supported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- graphql, javascript
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100