drizzle-team / drizzle-team/drizzle-orm
[BUG]: Vector similarity search example doesn't use index
- Dominant language
- TypeScript
- Stars
- 35.8k
- Forks
- 1.6k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 4
Description
### Report hasn't been filed before.
- [x] I have verified that the bug I'm about to report hasn't been filed before.
### What version of `drizzle-orm` are you using?
0.40.0
### What version of `drizzle-kit` are you using?
0.30.5
### Other packages
_No response_
### Describe the Bug
I'm following [the example](https://orm.drizzle.team/docs/guides/vector-similarity-search) however the index `index('embeddingIndex').using('hnsw', table.embedding.op('vector_cosine_ops'))` is not being used in the query
```
db
.select({ name: guides.title, url: guides.url, similarity })
.from(guides)
.where(gt(similarity, 0.5))
.orderBy((t) => desc(t.similarity))
.limit(4);
```
When you use explain-analyze, you get something like
```
Result {
command: 'EXPLAIN',
rowCount: null,
oid: null,
rows: [
{
'QUERY PLAN': 'Limit (cost=115124.73..115143.99 rows=4 width=48) (actual time=1897.720..1900.686 rows=4 loops=1)'
},
{
'QUERY PLAN': ' -> Nested Loop (cost=115124.73..411717.27 rows=61622 width=48) (actual time=1897.719..1900.683 rows=4 loops=1)'
},
{
'QUERY PLAN': ' -> Gather Merge (cost=115124.30..122301.20 rows=61622 width=1563) (actual time=1895.769..1895.852 rows=4 loops=1)'
},
{ 'QUERY PLAN': ' Workers Planned: 2' },
{ 'QUERY PLAN': ' Workers Launched: 0' },
{
'QUERY PLAN': ' -> Sort (cost=114124.28..114188.47 rows=25676 width=1563) (actual time=1895.048..1895.066 rows=4 loops=1)'
},
{
'QUERY PLAN': " Sort Key: (('1'::double precision - (table2.embedding <=> '[-0.111913756,-0.035390027,-0.085482165,-0.056525037,0.030906877,-0.040148906,0.032525755,0.06750378,-0.034749784,-0.103384905,-0.0006573094,0.062776044,-0.019352416,-0.032765888,-0.109152526,-0.042845387,-0.037449095,-0.015502359,0.012418087,0.03299091,-0.0076429355,0.008702347,0.020551344,0.011621866,-0.023128964,0.051614318,0.12912652,0.040404312,0.020590626,-0.09791204,0.08953167,0.070541196,-0.01741359,-0.024579301,0.07442026,-0.0396094,0.046302468,-0.0102278,0.008452137,-0.020843038,0.0067201904,-0.031880677,0.061711457,0.044971712,-0.035930194,0.016497206,-0.08907557,0.009295844,0.06919541,0.00835245,0.047864057,-0.042177387,-0.038789444,0.021467814,-0.02193836,0.07321112,-0.029600792,-0.0014396659,0.0010864731,-0.102234714,0.12582889,0.05554013,0.018099377,-0.015241772,0.019365313,0.00061216217,-0.0041250736,-0.05536567,0.024065802,-0.012238941,-0.010691057,0.049028277,-0.061153892,0.039460827,-0.010056771,0.07967286,0.04410146,-0.0051469486,0.094632946,-0.086514056,-0.003368785,-0.025152305,-0.015589282,0.02500768,0.033460185,0.059440106,0.007308837,0.015377941,0.031601682,-0.022664197,-0.009347118,-0.011111719,0.022178713,-0.05962877,-0.09179117,-0.038186703,0.07089183,0.053711053,-0.01400479,0.06721662,0.04174706,0.051942058,-0.065003484,0.013929912,-0.12759434,-0.03554935,-0.035142876,0.021790905,-0.02611605,-0.0481968,-0.05013993,0.054616272,0.024281234,-0.06634338,-0.036605023,-0.05997145,-0.120068125,-0.06558118,0.0019949276,-0.08021653,-0.029579166,0.006708046,-0.10176802,-0.05340342,0.0005364652,-0.023262814,0.026120944,5.966443e-33,-0.051135555,0.05016982,0.0051217554,0.034561146,0.008988026,0.00705136,-0.0073773293,-0.049493138,-0.07823088,0.075234935,-0.08592422,0.001550984,-0.047985617,-0.051809117,-0.08258111,-0.07854292,0.1013645,-0.03561708,-0.036745843,-0.03188112,-0.005059948,0.038823105,0.0106159095,0.014878499,0.017495511,0.039440837,-0.013541729,-0.020315595,0.039922085,0.034517698,0.05090319,-0.031118495,0.0068773194,0.013747903,-0.041043967,-0.0043250723,-0.05820994,-0.10716588,-0.00953306,-0.046458047,0.006226407,0.029235113,-0.059371974,0.018895948,-0.0008449653,0.026831178,0.054783426,0.08483516,0.12959337,0.035160795,-0.13360271,-0.041520353,-0.04503674,0.0050119474,0.0069712526,-0.025146812,-0.016877897,-0.031688903,-0.008304628,0.0068605957,0.03432457,0.104718864,-0.018131478,-0.054545358,0.05758843,-0.00094806025,0.04280037,-0.08460738,0.049713496,-0.083938174,-0.05791314,-0.017056221,0.099632576,0.024244381,0.04916464,-0.02128052,0.027253557,-0.03535373,-0.031928692,0.005589576,-0.082858436,-0.0051562013,-0.022246322,0.0711433,0.029550638,-0.034581877,-0.03422446,0.001908878,0.02391848,0.12584168,-0.017681006,-0.11355681,0.00793781,0.056528144,0.052189596,-4.8904466e-33,-0.093008645,-0.030977033,-0.038478293,0.033773184,-0.040083185,0.030240795,0.03334976,0.0020323042,-0.003443032,-0.07644613,0.05853363,0.044503424,0.009631041,-0.094412394,0.035707716,8.63719e-05,-0.004687876,-0.052315414,0.039439265,0.04349422,-0.028099455,0.097066514,-0.046370916,0.07240586,-0.022754768,0.07022007,0.07307134,0.06783445,0.0027630832,0.068968244,0.02551617,0.06127632,-0.009610611,0.11732342,-0.08506572,-0.088440135,0.13480452,0.077458605,-0.045417693,-0.08215468,-0.037424393,0.023379834,-0.02839842,0.01909581,-0.055125516,-0.035151675,0.00832898,-0.14673945,0.09646412,0.021242019,-0.0028001587,-0.034947425,0.020556146,0.043417122,0.07880786,0.04661198,0.017108334,0.04973708,0.04311942,0.03029299,0.12654027,0.058662444,0.08108261,0.06444462,0.036679465,-0.034349337,-0.020997696,0.045782384,0.054280784,-0.052825138,-0.00492721,0.051527552,0.0395326,0.049697425,0.03325899,-0.089628585,-0.021324705,-0.0060146907,-0.03971108,0.0019207825,-0.031234307,-0.014185364,-0.08700829,0.026964158,-0.02700664,-0.02096665,0.03663123,0.04308958,0.029336063,-0.04677498,-0.048523307,-0.01165394,-0.01572979,0.024062242,-0.02008861,-2.3156293e-08,-0.045433518,-0.0014223303,0.011440929,0.0035378544,0.004361558,-0.0016652504,0.026046012,0.05177778,-0.04103698,-0.038031768,0.073787466,-0.025825793,0.052464332,-0.055209897,-0.004655415,-0.04738716,-0.030137846,0.053788017,-0.026699694,-0.0680097,0.030421736,-0.009423589,0.010233486,-0.06691457,0.032074053,-0.007640074,0.042621672,0.12692733,0.038572464,-0.021430336,0.028470935,-0.012232111,0.094240025,-0.013012698,0.035121795,-0.03884423,-0.039491896,0.1005347,0.03694253,0.011489154,-0.013632421,0.044916205,-0.018028254,0.030338487,-0.038694516,-3.258202e-06,-0.052936275,0.04801675,0.015709966,0.0101912,0.028168093,-0.08488129,-0.010100336,-0.0006589756,0.017474927,-0.03629061,-0.01841257,0.051544487,-0.029709613,0.0030884994,-0.07196582,-0.0081340615,-0.01936272,-0.039696194]'::vector))) DESC"
},
{
'QUERY PLAN': ' Sort Method: quicksort Memory: 1319kB'
},
{
'QUERY PLAN': ' -> Parallel Seq Scan on table2 (cost=0.00..94778.75 rows=25676 width=1563) (actual time=4.843..1893.706 rows=833 loops=1)'
},
{
'QUERY PLAN': " Filter: ((embedding IS NOT NULL) AND (('1'::double precision - (embedding <=> '[-0.111913756,-0.035390027,-0.085482165,-0.056525037,0.030906877,-0.040148906,0.032525755,0.06750378,-0.034749784,-0.103384905,-0.0006573094,0.062776044,-0.019352416,-0.032765888,-0.109152526,-0.042845387,-0.037449095,-0.015502359,0.012418087,0.03299091,-0.0076429355,0.008702347,0.020551344,0.011621866,-0.023128964,0.051614318,0.12912652,0.040404312,0.020590626,-0.09791204,0.08953167,0.070541196,-0.01741359,-0.024579301,0.07442026,-0.0396094,0.046302468,-0.0102278,0.008452137,-0.020843038,0.0067201904,-0.031880677,0.061711457,0.044971712,-0.035930194,0.016497206,-0.08907557,0.009295844,0.06919541,0.00835245,0.047864057,-0.042177387,-0.038789444,0.021467814,-0.02193836,0.07321112,-0.029600792,-0.0014396659,0.0010864731,-0.102234714,0.12582889,0.05554013,0.018099377,-0.015241772,0.019365313,0.00061216217,-0.0041250736,-0.05536567,0.024065802,-0.012238941,-0.010691057,0.049028277,-0.061153892,0.039460827,-0.010056771,0.07967286,0.04410146,-0.0051469486,0.094632946,-0.086514056,-0.003368785,-0.025152305,-0.015589282,0.02500768,0.033460185,0.059440106,0.007308837,0.015377941,0.031601682,-0.022664197,-0.009347118,-0.011111719,0.022178713,-0.05962877,-0.09179117,-0.038186703,0.07089183,0.053711053,-0.01400479,0.06721662,0.04174706,0.051942058,-0.065003484,0.013929912,-0.12759434,-0.03554935,-0.035142876,0.021790905,-0.02611605,-0.0481968,-0.05013993,0.054616272,0.024281234,-0.06634338,-0.036605023,-0.05997145,-0.120068125,-0.06558118,0.0019949276,-0.08021653,-0.029579166,0.006708046,-0.10176802,-0.05340342,0.0005364652,-0.023262814,0.026120944,5.966443e-33,-0.051135555,0.05016982,0.0051217554,0.034561146,0.008988026,0.00705136,-0.0073773293,-0.049493138,-0.07823088,0.075234935,-0.08592422,0.001550984,-0.047985617,-0.051809117,-0.08258111,-0.07854292,0.1013645,-0.03561708,-0.036745843,-0.03188112,-0.005059948,0.038823105,0.0106159095,0.014878499,0.017495511,0.039440837,-0.013541729,-0.020315595,0.039922085,0.034517698,0.05090319,-0.031118495,0.0068773194,0.013747903,-0.041043967,-0.0043250723,-0.05820994,-0.10716588,-0.00953306,-0.046458047,0.006226407,0.029235113,-0.059371974,0.018895948,-0.0008449653,0.026831178,0.054783426,0.08483516,0.12959337,0.035160795,-0.13360271,-0.041520353,-0.04503674,0.0050119474,0.0069712526,-0.025146812,-0.016877897,-0.031688903,-0.008304628,0.0068605957,0.03432457,0.104718864,-0.018131478,-0.054545358,0.05758843,-0.00094806025,0.04280037,-0.08460738,0.049713496,-0.083938174,-0.05791314,-0.017056221,0.099632576,0.024244381,0.04916464,-0.02128052,0.027253557,-0.03535373,-0.031928692,0.005589576,-0.082858436,-0.0051562013,-0.022246322,0.0711433,0.029550638,-0.034581877,-0.03422446,0.001908878,0.02391848,0.12584168,-0.017681006,-0.11355681,0.00793781,0.056528144,0.052189596,-4.8904466e-33,-0.093008645,-0.030977033,-0.038478293,0.033773184,-0.040083185,0.030240795,0.03334976,0.0020323042,-0.003443032,-0.07644613,0.05853363,0.044503424,0.009631041,-0.094412394,0.035707716,8.63719e-05,-0.004687876,-0.052315414,0.039439265,0.04349422,-0.028099455,0.097066514,-0.046370916,0.07240586,-0.022754768,0.07022007,0.07307134,0.06783445,0.0027630832,0.068968244,0.02551617,0.06127632,-0.009610611,0.11732342,-0.08506572,-0.088440135,0.13480452,0.077458605,-0.045417693,-0.08215468,-0.037424393,0.023379834,-0.02839842,0.01909581,-0.055125516,-0.035151675,0.00832898,-0.14673945,0.09646412,0.021242019,-0.0028001587,-0.034947425,0.020556146,0.043417122,0.07880786,0.04661198,0.017108334,0.04973708,0.04311942,0.03029299,0.12654027,0.058662444,0.08108261,0.06444462,0.036679465,-0.034349337,-0.020997696,0.045782384,0.054280784,-0.052825138,-0.00492721,0.051527552,0.0395326,0.049697425,0.03325899,-0.089628585,-0.021324705,-0.0060146907,-0.03971108,0.0019207825,-0.031234307,-0.014185364,-0.08700829,0.026964158,-0.02700664,-0.02096665,0.03663123,0.04308958,0.029336063,-0.04677498,-0.048523307,-0.01165394,-0.01572979,0.024062242,-0.02008861,-2.3156293e-08,-0.045433518,-0.0014223303,0.011440929,0.0035378544,0.004361558,-0.0016652504,0.026046012,0.05177778,-0.04103698,-0.038031768,0.073787466,-0.025825793,0.052464332,-0.055209897,-0.004655415,-0.04738716,-0.030137846,0.053788017,-0.026699694,-0.0680097,0.030421736,-0.009423589,0.010233486,-0.06691457,0.032074053,-0.007640074,0.042621672,0.12692733,0.038572464,-0.021430336,0.028470935,-0.012232111,0.094240025,-0.013012698,0.035121795,-0.03884423,-0.039491896,0.1005347,0.03694253,0.011489154,-0.013632421,0.044916205,-0.018028254,0.030338487,-0.038694516,-3.258202e-06,-0.052936275,0.04801675,0.015709966,0.0101912,0.028168093,-0.08488129,-0.010100336,-0.0006589756,0.017474927,-0.03629061,-0.01841257,0.051544487,-0.029709613,0.0030884994,-0.07196582,-0.0081340615,-0.01936272,-0.039696194]'::vector)) > '0.5'::double precision))"
},
{
'QUERY PLAN': ' Rows Removed by Filter: 3684791'
},
{
'QUERY PLAN': ' -> Index Scan using table1_pkey on table1 (cost=0.43..4.69 rows=1 width=21) (actual time=1.192..1.192 rows=1 loops=4)'
},
{
'QUERY PLAN': ' Index Cond: (id = table2.profile_id)'
},
{ 'QUERY PLAN': 'Planning Time: 6.386 ms' },
{ 'QUERY PLAN': 'Execution Time: 1901.022 ms' }
],
fields: [
Field {
name: 'QUERY PLAN',
tableID: 0,
columnID: 0,
dataTypeID: 25,
dataTypeSize: -1,
dataTypeModifier: -1,
format: 'text'
}
],
_parsers: [ [Function: noParse] ],
_types: { getTypeParser: [Function: getTypeParser] },
RowCtor: null,
rowAsArray: false,
_prebuiltEmptyResultObject: { 'QUERY PLAN': null }
}
```
Not able to figure out how to make it use the index. Any help appreciated.
Contributor guide
Assessment
This issue has not been assessed yet.