quickwit-oss / quickwit-oss/quickwit
Double sorting with aggregation not working
Open
@PSeitz is already working on this.
Since Jun 13, 2024.
bug
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
When we request collection with aggregation with sorting by two fields we see two bugs:
- Missing sorting
- Different results for each call
Steps to reproduce (if applicable)
- Download dataset and index config from https://dev2.manticoresearch.com/index-settings-and-data.zip
- Run Quickwit in Docker
quickwit/quickwit:0.8.1 - Create index (config provided in attached archive):
export HOST='http://localhost:7280'
curl -s -XPOST "${HOST}/api/v1/indexes" \
--header "content-type: application/yaml" \
--data-binary @./index-config.yaml
- Upload data (Dataset is pretty big, so we split it into chunks):
split -l 10000 ./data.jsonl ./data_splitted.
echo "Starting loading"
for f in ./data_splitted.*; do
echo "Upload chunk $f"
curl -s -XPOST "${HOST}/api/v1/hn_small/ingest?commit=force" --data-binary @$f
rm $f
done
echo "Finished"
- Perform query:
curl --location '${HOST}/api/v1/hn_small/search' \
--header 'Content-Type: application/json' \
--data '{"query":"*","max_hits":0,"aggs":{"comment_ranking_avg":{"terms":{"field":"comment_ranking","size":20,"order":{"avg_field":"desc","_key":"desc"}},"aggs":{"avg_field":{"avg":{"field":"author_comment_count"}}}}}}'
- We got results with the wrong sorting
{
"num_hits": 1165439,
"hits": [],
"elapsed_time_micros": 6665,
"errors": [],
"aggregations": {
"comment_ranking_avg": {
"buckets": [
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 928.0 # Should be 2nd
},
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 961.0 # Should be 1st
},
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 730.0
},
.....
- if you repeat the request several times it can return different results (for the same query)
{
"num_hits": 1165439,
"hits": [],
"elapsed_time_micros": 9610,
"errors": [],
"aggregations": {
"comment_ranking_avg": {
"buckets": [
{
"avg_field": {
"value": 64.0
},
"doc_count": 1,
"key": 1305.0
},
{
"avg_field": {
"value": 117.0
},
"doc_count": 1,
"key": 1296.0
},
{
"avg_field": {
"value": 40.0
},
"doc_count": 1,
"key": 1289.0
},
{
"avg_field": {
"value": 87.0
},
"doc_count": 1,
"key": 1287.0
},
......
PS: Sometimes it returns results without grouping. In that case you should reindex your dataset
"buckets": [
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 961.0
},
{
"avg_field": {
"value": 3080.0
},
"doc_count": 1,
"key": 980.0
},
{
"avg_field": {
"value": 3077.0
},
"doc_count": 1,
"key": 1176.0
},
So generally we can get 3 different results for one query.
PS: Elasticsearch compatible URL has the same behaviour
Expected behavior
It should return the dataset like provided below
{
"num_hits": 1165439,
"hits": [],
"elapsed_time_micros": 6665,
"errors": [],
"aggregations": {
"comment_ranking_avg": {
"buckets": [
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 961.0
},
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 928.0
},
{
"avg_field": {
"value": 3504.0
},
"doc_count": 1,
"key": 730.0
},
.....
Configuration:
Please provide:
- Output of
quickwit --version
Quickwit 0.8.1 (aarch64-unknown-linux-gnu 2024-03-29T14:09:41Z e6c5396)
- The index_config.yaml
Provided in the attached archive)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.