HarperFast / HarperFast/harper

search_by_conditions can't do vector/k-NN search on @embed / HNSW-indexed columns

Open
#1,826 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A table using the `@embed` GraphQL directive gets an auto-generated `[Float]` embedding column with an HNSW index (`resources/graphql.ts`, `resources/models/embedHook.ts`). But that vector column can only be queried via the REST/GraphQL Query `sort` clause (`sort: { attribute, target, distance }` → `Table.search()` → `comparator: 'sort'` → HNSW `customIndex.search`), never through the operations-API `search_by_conditions` call.

This is an API-parity gap: everything else on a table (equality, range, prefix conditions) is reachable through both the Query interface and `search_by_conditions`, but nearest-neighbor/similarity search is Query-only.

## Root cause

- `validation/searchValidator.ts:42-55` — `comparator` is a fixed Joi enum (`equals`, `contains`, `starts_with`, `ends_with`, `greater_than[_equal]`, `less_than[_equal]`, `between`, `not_equal`); there's no vector/similarity comparator, and every condition's `value` is required (scalar).
- `dataLayer/harperBridge/ResourceBridge.ts` `mapCondition` (~lines 62-75) normalizes each condition down to `{ attribute, comparator, value }`, so even if a caller attaches a `target` vector + `distance` to a condition, it's dropped before reaching the index.
- `resources/indexes/HierarchicalNavigableSmallWorld.ts` `search()` (~lines 925-963) only accepts `sort`/`lt`/`le` comparators and expects `target`/`distance` — which the `search_by_conditions` path never constructs.
- The MCP tool schema for search operations (`components/mcp/tools/schemas/operations.ts:148`) has the same narrower scalar-only comparator enum.

## Proposal

Extend `search_by_conditions` to accept a vector search condition — e.g. a `sort`-comparator condition (or a dedicated `nearest`/`similar_to` comparator) carrying `target` (query vector) and `distance` (`cosine`/`euclidean`/`dot`), validated against columns backed by an HNSW custom index, and routed through the same `Table.search()` / `customIndex.search()` path the Query `sort` clause already uses. This would let operations-API/MCP callers do k-NN search on `@embed` tables without going through GraphQL/REST.

## Current workaround

Vector search today must go through the Query/GraphQL/REST `sort` clause or the programmatic `search()` Resource API (which also supports `vectorFilter`, JS-only) — see `integrationTests/qa-scratch/vector-search.test.ts` for the supported pattern.

Contributor guide

Open the contributing guide

Research direction

Read validation/searchValidator.ts and ResourceBridge.ts mapCondition first, then trace the existing sort path through Table.search() and resources/indexes/HierarchicalNavigableSmallWorld.ts. Check components/mcp/tools/schemas/operations.ts and integrationTests/qa-scratch/vector-search.test.ts. Done means operations-API and MCP callers can perform validated vector search on HNSW-backed columns with coverage for the supported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, javascript
Domain
api, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.