drizzle-team / drizzle-team/drizzle-orm

[BUG]: Vector similarity search example doesn't use index

Open
#4,398 0 comments 0 reactions 0 assignees View on GitHub
bug docs qb/crud
Dominant language
TypeScript
Stars
35.8k
Forks
1.6k
Avg merge
2d 7h
Merged PRs (30d)
4

Description

### Report hasn't been filed before.

- [x] I have verified that the bug I'm about to report hasn't been filed before.

### What version of `drizzle-orm` are you using?

0.40.0

### What version of `drizzle-kit` are you using?

0.30.5

### Other packages

_No response_

### Describe the Bug

I'm following [the example](https://orm.drizzle.team/docs/guides/vector-similarity-search) however the index `index('embeddingIndex').using('hnsw', table.embedding.op('vector_cosine_ops'))` is not being used in the query
```
db
.select({ name: guides.title, url: guides.url, similarity })
.from(guides)
.where(gt(similarity, 0.5))
.orderBy((t) => desc(t.similarity))
.limit(4);
```

When you use explain-analyze, you get something like
```
Result {
command: 'EXPLAIN',
rowCount: null,
oid: null,
rows: [
{
'QUERY PLAN': 'Limit (cost=115124.73..115143.99 rows=4 width=48) (actual time=1897.720..1900.686 rows=4 loops=1)'
},
{
'QUERY PLAN': ' -> Nested Loop (cost=115124.73..411717.27 rows=61622 width=48) (actual time=1897.719..1900.683 rows=4 loops=1)'
},
{
'QUERY PLAN': ' -> Gather Merge (cost=115124.30..122301.20 rows=61622 width=1563) (actual time=1895.769..1895.852 rows=4 loops=1)'
},
{ 'QUERY PLAN': ' Workers Planned: 2' },
{ 'QUERY PLAN': ' Workers Launched: 0' },
{
'QUERY PLAN': ' -> Sort (cost=114124.28..114188.47 rows=25676 width=1563) (actual time=1895.048..1895.066 rows=4 loops=1)'
},
{
'QUERY PLAN': " Sort Key: (('1'::double precision - (table2.embedding <=> '[-0.111913756,-0.035390027,-0.085482165,-0.056525037,0.030906877,-0.040148906,0.032525755,0.06750378,-0.034749784,-0.103384905,-0.0006573094,0.062776044,-0.019352416,-0.032765888,-0.109152526,-0.042845387,-0.037449095,-0.015502359,0.012418087,0.03299091,-0.0076429355,0.008702347,0.020551344,0.011621866,-0.023128964,0.051614318,0.12912652,0.040404312,0.020590626,-0.09791204,0.08953167,0.070541196,-0.01741359,-0.024579301,0.07442026,-0.0396094,0.046302468,-0.0102278,0.008452137,-0.020843038,0.0067201904,-0.031880677,0.061711457,0.044971712,-0.035930194,0.016497206,-0.08907557,0.009295844,0.06919541,0.00835245,0.047864057,-0.042177387,-0.038789444,0.021467814,-0.02193836,0.07321112,-0.029600792,-0.0014396659,0.0010864731,-0.102234714,0.12582889,0.05554013,0.018099377,-0.015241772,0.019365313,0.00061216217,-0.0041250736,-0.05536567,0.024065802,-0.012238941,-0.010691057,0.049028277,-0.061153892,0.039460827,-0.010056771,0.07967286,0.04410146,-0.0051469486,0.094632946,-0.086514056,-0.003368785,-0.025152305,-0.015589282,0.02500768,0.033460185,0.059440106,0.007308837,0.015377941,0.031601682,-0.022664197,-0.009347118,-0.011111719,0.022178713,-0.05962877,-0.09179117,-0.038186703,0.07089183,0.053711053,-0.01400479,0.06721662,0.04174706,0.051942058,-0.065003484,0.013929912,-0.12759434,-0.03554935,-0.035142876,0.021790905,-0.02611605,-0.0481968,-0.05013993,0.054616272,0.024281234,-0.06634338,-0.036605023,-0.05997145,-0.120068125,-0.06558118,0.0019949276,-0.08021653,-0.029579166,0.006708046,-0.10176802,-0.05340342,0.0005364652,-0.023262814,0.026120944,5.966443e-33,-0.051135555,0.05016982,0.0051217554,0.034561146,0.008988026,0.00705136,-0.0073773293,-0.049493138,-0.07823088,0.075234935,-0.08592422,0.001550984,-0.047985617,-0.051809117,-0.08258111,-0.07854292,0.1013645,-0.03561708,-0.036745843,-0.03188112,-0.005059948,0.038823105,0.0106159095,0.014878499,0.017495511,0.039440837,-0.013541729,-0.020315595,0.039922085,0.034517698,0.05090319,-0.031118495,0.0068773194,0.013747903,-0.041043967,-0.0043250723,-0.05820994,-0.10716588,-0.00953306,-0.046458047,0.006226407,0.029235113,-0.059371974,0.018895948,-0.0008449653,0.026831178,0.054783426,0.08483516,0.12959337,0.035160795,-0.13360271,-0.041520353,-0.04503674,0.0050119474,0.0069712526,-0.025146812,-0.016877897,-0.031688903,-0.008304628,0.0068605957,0.03432457,0.104718864,-0.018131478,-0.054545358,0.05758843,-0.00094806025,0.04280037,-0.08460738,0.049713496,-0.083938174,-0.05791314,-0.017056221,0.099632576,0.024244381,0.04916464,-0.02128052,0.027253557,-0.03535373,-0.031928692,0.005589576,-0.082858436,-0.0051562013,-0.022246322,0.0711433,0.029550638,-0.034581877,-0.03422446,0.001908878,0.02391848,0.12584168,-0.017681006,-0.11355681,0.00793781,0.056528144,0.052189596,-4.8904466e-33,-0.093008645,-0.030977033,-0.038478293,0.033773184,-0.040083185,0.030240795,0.03334976,0.0020323042,-0.003443032,-0.07644613,0.05853363,0.044503424,0.009631041,-0.094412394,0.035707716,8.63719e-05,-0.004687876,-0.052315414,0.039439265,0.04349422,-0.028099455,0.097066514,-0.046370916,0.07240586,-0.022754768,0.07022007,0.07307134,0.06783445,0.0027630832,0.068968244,0.02551617,0.06127632,-0.009610611,0.11732342,-0.08506572,-0.088440135,0.13480452,0.077458605,-0.045417693,-0.08215468,-0.037424393,0.023379834,-0.02839842,0.01909581,-0.055125516,-0.035151675,0.00832898,-0.14673945,0.09646412,0.021242019,-0.0028001587,-0.034947425,0.020556146,0.043417122,0.07880786,0.04661198,0.017108334,0.04973708,0.04311942,0.03029299,0.12654027,0.058662444,0.08108261,0.06444462,0.036679465,-0.034349337,-0.020997696,0.045782384,0.054280784,-0.052825138,-0.00492721,0.051527552,0.0395326,0.049697425,0.03325899,-0.089628585,-0.021324705,-0.0060146907,-0.03971108,0.0019207825,-0.031234307,-0.014185364,-0.08700829,0.026964158,-0.02700664,-0.02096665,0.03663123,0.04308958,0.029336063,-0.04677498,-0.048523307,-0.01165394,-0.01572979,0.024062242,-0.02008861,-2.3156293e-08,-0.045433518,-0.0014223303,0.011440929,0.0035378544,0.004361558,-0.0016652504,0.026046012,0.05177778,-0.04103698,-0.038031768,0.073787466,-0.025825793,0.052464332,-0.055209897,-0.004655415,-0.04738716,-0.030137846,0.053788017,-0.026699694,-0.0680097,0.030421736,-0.009423589,0.010233486,-0.06691457,0.032074053,-0.007640074,0.042621672,0.12692733,0.038572464,-0.021430336,0.028470935,-0.012232111,0.094240025,-0.013012698,0.035121795,-0.03884423,-0.039491896,0.1005347,0.03694253,0.011489154,-0.013632421,0.044916205,-0.018028254,0.030338487,-0.038694516,-3.258202e-06,-0.052936275,0.04801675,0.015709966,0.0101912,0.028168093,-0.08488129,-0.010100336,-0.0006589756,0.017474927,-0.03629061,-0.01841257,0.051544487,-0.029709613,0.0030884994,-0.07196582,-0.0081340615,-0.01936272,-0.039696194]'::vector))) DESC"
},
{
'QUERY PLAN': ' Sort Method: quicksort Memory: 1319kB'
},
{
'QUERY PLAN': ' -> Parallel Seq Scan on table2 (cost=0.00..94778.75 rows=25676 width=1563) (actual time=4.843..1893.706 rows=833 loops=1)'
},
{
'QUERY PLAN': " Filter: ((embedding IS NOT NULL) AND (('1'::double precision - (embedding <=> '[-0.111913756,-0.035390027,-0.085482165,-0.056525037,0.030906877,-0.040148906,0.032525755,0.06750378,-0.034749784,-0.103384905,-0.0006573094,0.062776044,-0.019352416,-0.032765888,-0.109152526,-0.042845387,-0.037449095,-0.015502359,0.012418087,0.03299091,-0.0076429355,0.008702347,0.020551344,0.011621866,-0.023128964,0.051614318,0.12912652,0.040404312,0.020590626,-0.09791204,0.08953167,0.070541196,-0.01741359,-0.024579301,0.07442026,-0.0396094,0.046302468,-0.0102278,0.008452137,-0.020843038,0.0067201904,-0.031880677,0.061711457,0.044971712,-0.035930194,0.016497206,-0.08907557,0.009295844,0.06919541,0.00835245,0.047864057,-0.042177387,-0.038789444,0.021467814,-0.02193836,0.07321112,-0.029600792,-0.0014396659,0.0010864731,-0.102234714,0.12582889,0.05554013,0.018099377,-0.015241772,0.019365313,0.00061216217,-0.0041250736,-0.05536567,0.024065802,-0.012238941,-0.010691057,0.049028277,-0.061153892,0.039460827,-0.010056771,0.07967286,0.04410146,-0.0051469486,0.094632946,-0.086514056,-0.003368785,-0.025152305,-0.015589282,0.02500768,0.033460185,0.059440106,0.007308837,0.015377941,0.031601682,-0.022664197,-0.009347118,-0.011111719,0.022178713,-0.05962877,-0.09179117,-0.038186703,0.07089183,0.053711053,-0.01400479,0.06721662,0.04174706,0.051942058,-0.065003484,0.013929912,-0.12759434,-0.03554935,-0.035142876,0.021790905,-0.02611605,-0.0481968,-0.05013993,0.054616272,0.024281234,-0.06634338,-0.036605023,-0.05997145,-0.120068125,-0.06558118,0.0019949276,-0.08021653,-0.029579166,0.006708046,-0.10176802,-0.05340342,0.0005364652,-0.023262814,0.026120944,5.966443e-33,-0.051135555,0.05016982,0.0051217554,0.034561146,0.008988026,0.00705136,-0.0073773293,-0.049493138,-0.07823088,0.075234935,-0.08592422,0.001550984,-0.047985617,-0.051809117,-0.08258111,-0.07854292,0.1013645,-0.03561708,-0.036745843,-0.03188112,-0.005059948,0.038823105,0.0106159095,0.014878499,0.017495511,0.039440837,-0.013541729,-0.020315595,0.039922085,0.034517698,0.05090319,-0.031118495,0.0068773194,0.013747903,-0.041043967,-0.0043250723,-0.05820994,-0.10716588,-0.00953306,-0.046458047,0.006226407,0.029235113,-0.059371974,0.018895948,-0.0008449653,0.026831178,0.054783426,0.08483516,0.12959337,0.035160795,-0.13360271,-0.041520353,-0.04503674,0.0050119474,0.0069712526,-0.025146812,-0.016877897,-0.031688903,-0.008304628,0.0068605957,0.03432457,0.104718864,-0.018131478,-0.054545358,0.05758843,-0.00094806025,0.04280037,-0.08460738,0.049713496,-0.083938174,-0.05791314,-0.017056221,0.099632576,0.024244381,0.04916464,-0.02128052,0.027253557,-0.03535373,-0.031928692,0.005589576,-0.082858436,-0.0051562013,-0.022246322,0.0711433,0.029550638,-0.034581877,-0.03422446,0.001908878,0.02391848,0.12584168,-0.017681006,-0.11355681,0.00793781,0.056528144,0.052189596,-4.8904466e-33,-0.093008645,-0.030977033,-0.038478293,0.033773184,-0.040083185,0.030240795,0.03334976,0.0020323042,-0.003443032,-0.07644613,0.05853363,0.044503424,0.009631041,-0.094412394,0.035707716,8.63719e-05,-0.004687876,-0.052315414,0.039439265,0.04349422,-0.028099455,0.097066514,-0.046370916,0.07240586,-0.022754768,0.07022007,0.07307134,0.06783445,0.0027630832,0.068968244,0.02551617,0.06127632,-0.009610611,0.11732342,-0.08506572,-0.088440135,0.13480452,0.077458605,-0.045417693,-0.08215468,-0.037424393,0.023379834,-0.02839842,0.01909581,-0.055125516,-0.035151675,0.00832898,-0.14673945,0.09646412,0.021242019,-0.0028001587,-0.034947425,0.020556146,0.043417122,0.07880786,0.04661198,0.017108334,0.04973708,0.04311942,0.03029299,0.12654027,0.058662444,0.08108261,0.06444462,0.036679465,-0.034349337,-0.020997696,0.045782384,0.054280784,-0.052825138,-0.00492721,0.051527552,0.0395326,0.049697425,0.03325899,-0.089628585,-0.021324705,-0.0060146907,-0.03971108,0.0019207825,-0.031234307,-0.014185364,-0.08700829,0.026964158,-0.02700664,-0.02096665,0.03663123,0.04308958,0.029336063,-0.04677498,-0.048523307,-0.01165394,-0.01572979,0.024062242,-0.02008861,-2.3156293e-08,-0.045433518,-0.0014223303,0.011440929,0.0035378544,0.004361558,-0.0016652504,0.026046012,0.05177778,-0.04103698,-0.038031768,0.073787466,-0.025825793,0.052464332,-0.055209897,-0.004655415,-0.04738716,-0.030137846,0.053788017,-0.026699694,-0.0680097,0.030421736,-0.009423589,0.010233486,-0.06691457,0.032074053,-0.007640074,0.042621672,0.12692733,0.038572464,-0.021430336,0.028470935,-0.012232111,0.094240025,-0.013012698,0.035121795,-0.03884423,-0.039491896,0.1005347,0.03694253,0.011489154,-0.013632421,0.044916205,-0.018028254,0.030338487,-0.038694516,-3.258202e-06,-0.052936275,0.04801675,0.015709966,0.0101912,0.028168093,-0.08488129,-0.010100336,-0.0006589756,0.017474927,-0.03629061,-0.01841257,0.051544487,-0.029709613,0.0030884994,-0.07196582,-0.0081340615,-0.01936272,-0.039696194]'::vector)) > '0.5'::double precision))"
},
{
'QUERY PLAN': ' Rows Removed by Filter: 3684791'
},
{
'QUERY PLAN': ' -> Index Scan using table1_pkey on table1 (cost=0.43..4.69 rows=1 width=21) (actual time=1.192..1.192 rows=1 loops=4)'
},
{
'QUERY PLAN': ' Index Cond: (id = table2.profile_id)'
},
{ 'QUERY PLAN': 'Planning Time: 6.386 ms' },
{ 'QUERY PLAN': 'Execution Time: 1901.022 ms' }
],
fields: [
Field {
name: 'QUERY PLAN',
tableID: 0,
columnID: 0,
dataTypeID: 25,
dataTypeSize: -1,
dataTypeModifier: -1,
format: 'text'
}
],
_parsers: [ [Function: noParse] ],
_types: { getTypeParser: [Function: getTypeParser] },
RowCtor: null,
rowAsArray: false,
_prebuiltEmptyResultObject: { 'QUERY PLAN': null }
}
```

Not able to figure out how to make it use the index. Any help appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.