tursodatabase / tursodatabase/libsql
vector_top_k() fails with query vector from CTE
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 17.2k
- Forks
- 531
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 1
Description
I'm getting a vector_top_k.xBestIndex malfunction error when using a query vector defined in a CTE.
My use case is that I want to return the distance with nearest neighbor data. vector_top_k() only returns id and not distance, so I need to call the distance function. I didn't want to repeat the query vector for both vector_distance_cos() and vector_top_k(), so I tried to define it in a CTE.
Here's a simple reproduction with the movies example:
-- This is the example table and index definition.
CREATE TABLE movies (
title TEXT,
year INT,
embedding F32_BLOB(4) -- 4-dimensional f32 vector
);
INSERT INTO movies (title, year, embedding)
VALUES
('Napoleon', 2023, vector32('[0.800, 0.579, 0.481, 0.229]')),
('Black Hawk Down', 2001, vector32('[0.406, 0.027, 0.378, 0.056]')),
('Gladiator', 2000, vector32('[0.698, 0.140, 0.073, 0.125]')),
('Blade Runner', 1982, vector32('[0.379, 0.637, 0.011, 0.647]'));
CREATE INDEX movies_idx ON movies(libsql_vector_idx(embedding));
-- This is the CTE that fails.
WITH common AS (SELECT vector32('[0.064, 0.777, 0.661, 0.687]') AS q)
SELECT title, year, vector_distance_cos(embedding, q)
FROM vector_top_k('movies_idx', q, 3)
JOIN movies ON movies.rowid = id
JOIN common
WHERE year >= 2020;
-- This similar CTE works without vector_top_k().
WITH common AS (SELECT vector32('[0.064, 0.777, 0.661, 0.687]') AS q)
SELECT title, year, vector_distance_cos(embedding, q)
FROM movies, common
WHERE year >= 2020;
Here's the failing statement on v0.24.31:
libsql> WITH common AS (SELECT vector32('[0.064, 0.777, 0.661, 0.687]') AS q)
SELECT title, year, vector_distance_cos(embedding, q)
FROM vector_top_k('movies_idx', q, 3)
JOIN movies ON movies.rowid = id
JOIN common
WHERE year >= 2020;
...> ...> ...> ...> ...> Parse error: vector_top_k.xBestIndex malfunction
libsql>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied CTE reproduction on v0.24.31 and compare it with the similar query that works without vector_top_k(). Trace the vector_top_k() path producing the “xBestIndex malfunction” error; done means the CTE query parses and returns nearest-neighbor rows with vector_distance_cos values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100