manticoresoftware / manticoresoftware/columnar

k=100K, but much less is returned

Open
#73 0 comments 0 reactions 1 assignee View on GitHub

@glookka is already working on this.

Since Dec 9, 2024.

bug est::size_S
Dominant language
C++
Stars
110
Forks
23
Avg merge
3d 5h
Merged PRs (30d)
8

Description

### Bug Description:

Take this file:
```
snikolaev@dev2:~$ ls -lah /home/snikolaev/lisdocument1.sql.gz
-rw-rw-r-- 1 snikolaev snikolaev 362M Dec 8 18:02 /home/snikolaev/lisdocument1.sql.gz
```

Load 100K docs with embeddings to Manticore:
```
snikolaev@dev2:/tmp$ time gzip -cd lisdocument1.sql.gz | mysql -P9306 -h0

real 0m53.528s
user 0m21.327s
sys 0m1.000s
```

There're 2 disk chunks + a ram chunk now. Merge to one disk chunk:
```
mysql> flush ramchunk lisdocument1;
mysql> optimize table lisdocument1 option cutoff=1, sync=1;
```

And the following query now returns ~80K docs even though the `k` is 100K and all the limits are also set to 100K and the `cutoff` is off. And sometimes it returns significantly less:

```
snikolaev@dev2:/tmp$ for n in `seq 1 10`; do gzip -cd lisdocument1.sql.gz | mysql -P9306 -h0; mysql -P9306 -h0 -e "flush ramchunk lisdocument1; optimize table lisdocument1 option sync=1, cutoff=1; SELECT count(*) FROM lisdocument1 WHERE knn (vector, 100000, (0.31006125,-0.0003656157,0.081459515,0.515771,0.014484076,-0.66133505,0.32237425,-0.55777,-0.386141,-0.023305604,0.11610652,-0.67190164,-0.08708035,0.08902938,-0.6518018,-0.051225036,-1.1461623,-0.41078997,0.17181408,-0.4735955,0.03203327,0.23325141,-0.9909842,-0.3320622,0.201117,0.6546775,-0.7582774,0.9626067,0.7869749,0.7003234,0.5161592,-0.47969586,1.1994319,0.28759456,-0.26597247,-0.22219379,0.33459783,-0.97726834,-0.4739887,0.43206725,0.01194832,-0.05791433,0.4811258,-0.8271685,0.3951732,0.24713722,0.11192199,1.0213652,0.19014879,-0.6778019,0.50749946,-1.049119,0.4575615,0.9339654,0.58290714,-0.097028166,-0.48701167,-1.0065265,-0.13599022,-0.40211567,-0.5544309,0.1297585,0.19523889,-0.77102447,0.64259815,-0.30415413,-0.23215108,0.67778605,0.618211,0.7600791,0.23732586,-0.7648567,0.062301278,0.13431896,0.66374356,0.40523198,0.46663165,0.51943946,-0.73183507,-0.57254523,0.50741565,-0.14492488,-0.026737308,-0.63081896,-0.041068904,0.43559575,0.1266691,0.68689376,0.013397118,0.53886193,0.20262197,0.29941437,-0.6999152,0.28189236,0.9438162,0.6245068,-0.2915007,-0.08586646,-0.39370182,0.6360048,-0.38291323,0.7843485,-0.54510206,-0.52077717,1.2116083,-0.12180229,0.13172503,-0.10871028,0.41399533,-0.47680008,0.9621055,-0.54643613,0.21518388,-0.54834914,-0.1906337,-0.09179794,-0.590677,-0.52437264,-0.7151974,-0.02695119,0.21971804,-0.024934726,0.05605529,0.44665658,-0.12020654,0.26943594,-0.10920261,0.20266409,0.012066222,0.19158714,-0.12457689,-0.4856409,0.16108908,-0.20201308,0.041395552,-0.34803805,-1.1102498,-0.74369335,-1.6364297,1.0657281,0.07174268,0.010433137,0.7891052,0.60267335,-0.2899389,0.4021704,-0.20868495,-0.003395542,-0.05169468,-0.17775702,0.21387385,0.0991693,-0.9152813,0.19724046,0.35551172,-0.4999967,-0.8879677,0.16530086,-1.1837602,-0.59362537,0.12462508,-0.34771776,0.9998891,0.8668287,-0.6882197,0.2541907,0.49837124,0.15233028,0.336384,-0.038960453,-0.7221209,-0.071910635,0.04892005,-0.21825159,0.16531892,-0.8357419,0.5070645,0.1569018,0.313003,-0.17091025,-0.33268282,1.1826873,0.24373363,-0.15638427,0.27422035,0.39059705,-0.34179664,-0.35423273,-0.7033785,0.21909295,1.1341084,0.27519116,1.1161767,-0.56932676,-0.9429059,0.40243575,-0.5576759,-0.21010822,-0.4341065,1.2042234,-0.82608193,0.4289299,-0.14796726,-0.027835188,-0.26135963,-0.5276095,-0.88423705,0.04380098,-0.43928683,0.41062203,-0.34576598,0.13919765,-0.01392624,-0.5553707,-0.45019755,0.91757697,0.4029668,0.1176647,-0.15419589,0.73095477,0.40260544,0.43138027,0.21548833,-0.5267257,-0.30721363,-0.25096813,-0.6048828,0.7789119,0.15235308,0.17003383,0.8087661,-0.38634503,-0.29575384,0.3041153,-0.0025674396,-0.12190996,0.5652693,0.123609096,-0.41633,-0.075612135,0.5379093,0.47812945,-0.7582835,0.48872802,0.17062183,-0.41053674,0.24030015,0.43435127,0.40152794,0.006724952,-0.32310417,-0.35114315,-0.37304744,-0.53361934,-0.14794493,0.5957107,0.48854348,0.22275516,0.28737432,-0.55769044,0.19703211,-0.43264365,0.70282584,0.8103258,0.4166922,0.18818198,-0.6881402,-0.8271788,0.33227536,0.44830915,-0.21777764,-0.3315875,-0.18762241,-0.5730435,1.2522116,0.05436271,0.5053515,0.08252778,-0.7671772,-0.67160285,0.7309791,-0.6450694,-0.23141778,0.081859656,-0.31854483,-0.66024923,0.14364149,0.30650187,0.14491636,0.09296628,0.29341727,-0.28662416,-0.8733087,0.73710185,0.6832434,-1.0337256,-0.36529627,-1.185272,0.70611227,-0.21201886,-0.4519033,0.35380524,0.24411415,0.7839299,-1.0546187,-0.5108705,-0.040125087,0.0038510442,0.16296358,0.14024405,-0.29560274,-0.36276716,1.3071771,0.086237356,-0.791835,0.11178309,-0.12588637,-0.4115165,0.30388063,-0.51085454,-0.090283126,-1.5626551,0.53435034,-0.22315815,-0.31446335,0.48053166,0.64726245,-1.073468,0.56286776,0.10031773,0.39174485,0.5910364,-0.27479896,-0.25149876,-0.064312816,-0.07417552,0.8132626,0.07798847,0.25687593,-0.60261244,-0.45493522,0.54915696,-0.053107496,-1.2380457,-0.21066967,-0.1940282,-0.43841827,0.17368084,-0.14619267,-0.2789112,0.2749565,0.4712215,0.3926232,-0.32814994,-1.0122043,-1.0696679,-0.16326195,-0.975733,0.39950857,0.19773667,-0.16035733,-0.058070622,-0.118425414,0.18218455,0.6603359,0.54957753,0.44137517,0.21397805,-0.22320518,-0.91974175,0.48958048,-0.084736824,-0.426293,0.8708147,-0.2713833,-0.39039102,0.30332446,-0.7585659,-0.12065592,-0.15349118,0.8288996,0.23025775,-0.35495567,0.21538107,0.86950296,-0.00998487,0.0017167316,0.120606296,0.4414424,0.4505997,0.829459,0.7318415,-0.8386215,0.2715878,0.65265125,-0.07971094,0.48587146,-0.5186518,0.55407375,0.2426369,0.21575326,-0.044891708,-1.2336338,-0.10642625,-0.5177275,-0.6177509,0.8570399,1.043228,-0.2713039,0.29890174,-0.47719136,0.09048428,0.81275654,0.22632912,-0.27022663,0.10290417,0.4140533,0.17572246,0.48987168,0.6703758,-0.46391836,0.075863644,15.550138,0.62152815,-0.04971984,0.10588257,0.17923146,0.15616268,-0.46352684,-0.08553317,-0.8809511,-0.13167156,0.88224876,-0.1254104,0.08551519,0.41633347,-0.39136684,-0.5745892,0.059034057,0.27425095,0.6071214,-1.021161,0.40765795,0.41800106,-0.2145601,0.020918911,0.60376966,0.48886603,0.23570696,-0.07289946,0.7897558,-0.38197488,0.8834917,-0.48737484,0.72564256,1.0258603,-0.7900168,-0.1589608,-0.5894068,-1.2211249,0.21977632,-0.032760903,-0.73974997,0.027299361,-0.40298098,-0.59684885,0.18560478,0.22460128,-0.25463927,0.6956338,0.04916962,-0.05426601,0.4170774,0.4209522,0.69340205,-0.5526944,0.13108079,0.028241986,0.27075502,0.058803745,-0.12544855,-0.6035113,-1.2496368,-0.6891652,-0.21734844,0.88274825,0.21550162,0.900132,0.23280652,0.022997925,0.070107326,0.2929837,-0.9167534,-0.44679806,0.6751843,0.26168746,-0.9102508,-0.53386253,-0.11613186,-0.33997464,-0.65545017,0.34133926,0.5415031,-0.8402537,-0.4420601,0.9791182,0.3558302,-1.3223187,0.58805984,-0.9981405,-1.0075716,-0.10522475,-0.4694484,-0.88297,-0.073731676,-0.40783596,-0.44489437,-0.35490802,0.5813389,-0.48236555,-0.8025031,0.14774305,-0.07733792,-0.04009475,-0.19666764,-0.13940135,0.56877077,0.89890414,-0.34569648,0.79776573,-0.091007896,-0.69942564,-1.1672308,0.30055726,-0.84184074,-1.0750866,0.53956366,-0.9331158,-0.29834008,0.0052645435,0.44171295,-0.4195513,0.8877479,-0.058045257,0.05742934,-0.16210446,-1.0204247,0.44277644,0.13323916,-1.1805303,-0.21072462,0.12990586,-0.044168193,-1.2782251,-0.25332808,0.40048394,0.2744026,-0.6884139,0.75036937,-0.8379228,0.1622562,0.1323186,-0.027955009,-0.42298132,-0.17523514,-0.64205575,0.44626942,0.11984201,0.61406684,-0.5398223,0.13014658,1.4530864,0.072602026,-0.34999475,-0.7632572,-0.3052821,-0.3685159,-0.30019602,-0.26202905,0.065592654,0.1356203,-0.23066288,-0.06168315,1.18814,0.18889207,-0.496132,0.71889156,0.3918985,-0.1458961,0.12719785,-0.8727122,-0.33598074,-0.20535125,-0.48016322,-0.40368298,-0.5138646,-0.62812066,0.455146,-0.23726645,-0.35210988,0.6264089,-0.35540456,-0.8420332,-0.4773195,-0.043095402,0.5031531,0.31352416,-0.56513035,0.6253261,-0.22605664,-0.022685153,0.56507564,0.13309203,-0.2219282,-0.74726635,-1.0282854,0.097224504,-0.030685902,-0.053559277,-1.3350763,0.6840824,-0.21796289,0.54228777,-0.1751819,0.185303,-1.3930091,0.28261814,0.55813426,0.058485728,-0.2205351,0.5061636,-0.45415583,-0.41728288,0.21399207,0.10460134,-0.5251125,-0.05460708,0.12028494,-0.8888363,-0.5616349,1.0626166,-0.57732487,-0.71718526,-0.3703574,0.18429625,0.5347522,0.20444115,0.57214594,0.37266445,-1.2059611,0.8612643,0.48821098,-0.43664938,-0.82340217,0.67264736,0.22491321,-0.74951804,0.0871341,-0.27833295,0.46309146,-0.78231233,-0.21181802,0.42298067,-0.07849596,0.5937467,1.0871675,0.019744089,-0.7014219,0.20075509,-0.64872456,-0.6484867,-0.48085892,0.7048896,0.37797463,-0.19823772,0.009384685,-0.44905084,0.47751105,0.07683447,-0.7515542,-0.14501087,0.17405201,0.17612083,0.54188806,-0.2535289,0.3439073,-0.6030962,0.38111544,-0.651172,-0.96390456,-0.030525658,-0.26229542,-0.07045684,0.28682828,0.4695437,0.48653972,-0.021991393,-0.19629684,-0.104265794,0.72771174,1.1777867,-0.42338756,-0.28041443,0.9536649,1.531987,-0.29626778,0.4382578,-0.715345,-0.8170925,0.94427216,1.243451,0.17655466,0.62224877,0.20158736,0.16209148,0.09094697,0.054510932,0.18709274,0.43218136,0.750771,0.52881354,-0.05870044,0.15825632,0.85584885,0.435063,0.10643143,0.95394486,1.6185488,-0.46994826,-0.0854546,-0.5466821,0.57927006,-0.428444,0.26448515,-0.6947298,1.218788,-0.14955972,0.18937686,0.12341008,-1.0414394,1.0318452,0.18007237,0.32144284,-0.41614947,-0.08107099,-0.24519788,-0.10084439,-0.5396372,0.36262137,-0.26675776,-0.21520412,0.14965774,-0.058672633,0.50249416,0.24154651,0.49916118,0.3448778,0.12129112,0.23797168,-1.1313815,-0.55008906,-0.7546291,-0.34192833,-1.1170193,-0.27139792,-1.0538054,-0.16249998,-0.90182513)) limit 100000 OPTION cutoff = 0, max_matches = 100000;"; done;

+----------+
| count(*) |
+----------+
| 82176 |
+----------+
+----------+
| count(*) |
+----------+
| 80652 |
+----------+
+----------+
| count(*) |
+----------+
| 79 |
+----------+
+----------+
| count(*) |
+----------+
| 82176 |
+----------+
+----------+
| count(*) |
+----------+
| 81921 |
+----------+
+----------+
| count(*) |
+----------+
| 82176 |
+----------+
+----------+
| count(*) |
+----------+
| 1224 |
+----------+
+----------+
| count(*) |
+----------+
| 82176 |
+----------+
+----------+
| count(*) |
+----------+
| 82176 |
+----------+
+----------+
| count(*) |
+----------+
| 82176 |
+----------+
```

### Notes
* Reducing the docs count to 10K solves both issues.

### Manticore Search Version:

Manticore 6.3.9 dd29aca47@24120423 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219) (knn 2.3.1 edadc69@24112219)

### Operating System Version:

Ubuntu Jammy (dev2)

### Have you tried the latest development version?

No

### Internal Checklist:

To be completed by the assignee. Check off tasks that have been completed or are not applicable.

- [ ] Implementation completed
- [ ] Tests developed
- [ ] Documentation updated
- [ ] Documentation reviewed
- [ ] Changelog updated

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.