quickwit-oss / quickwit-oss/quickwit

Double sorting with aggregation not working

Open
#5,120 4 comments 0 reactions 1 assignee View on GitHub

@PSeitz is already working on this.

Since Jun 13, 2024.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug

When we request collection with aggregation with sorting by two fields we see two bugs:

  • Missing sorting
  • Different results for each call

Steps to reproduce (if applicable)

  1. Download dataset and index config from https://dev2.manticoresearch.com/index-settings-and-data.zip
  2. Run Quickwit in Docker quickwit/quickwit:0.8.1
  3. Create index (config provided in attached archive):
export HOST='http://localhost:7280'

curl -s -XPOST "${HOST}/api/v1/indexes" \
    --header "content-type: application/yaml" \
    --data-binary @./index-config.yaml
  1. Upload data (Dataset is pretty big, so we split it into chunks):
split -l 10000 ./data.jsonl ./data_splitted.

echo "Starting loading"
for f in ./data_splitted.*; do
    echo "Upload chunk $f"
    curl -s -XPOST "${HOST}/api/v1/hn_small/ingest?commit=force" --data-binary @$f
    rm $f
done
echo "Finished"
  1. Perform query:
curl --location '${HOST}/api/v1/hn_small/search' \
--header 'Content-Type: application/json' \
--data '{"query":"*","max_hits":0,"aggs":{"comment_ranking_avg":{"terms":{"field":"comment_ranking","size":20,"order":{"avg_field":"desc","_key":"desc"}},"aggs":{"avg_field":{"avg":{"field":"author_comment_count"}}}}}}'
  1. We got results with the wrong sorting
{
    "num_hits": 1165439,
    "hits": [],
    "elapsed_time_micros": 6665,
    "errors": [],
    "aggregations": {
        "comment_ranking_avg": {
            "buckets": [
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key": 928.0 # Should be 2nd
                },
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key": 961.0 # Should be 1st
                },
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key": 730.0
                },
.....                
  1. if you repeat the request several times it can return different results (for the same query)
{
    "num_hits": 1165439,
    "hits": [],
    "elapsed_time_micros": 9610,
    "errors": [],
    "aggregations": {
        "comment_ranking_avg": {
            "buckets": [
                {
                    "avg_field": {
                        "value": 64.0
                    },
                    "doc_count": 1,
                    "key": 1305.0
                },
                {
                    "avg_field": {
                        "value": 117.0
                    },
                    "doc_count": 1,
                    "key": 1296.0
                },
                {
                    "avg_field": {
                        "value": 40.0
                    },
                    "doc_count": 1,
                    "key": 1289.0
                },
                {
                    "avg_field": {
                        "value": 87.0
                    },
                    "doc_count": 1,
                    "key": 1287.0
                },
......

PS: Sometimes it returns results without grouping. In that case you should reindex your dataset

"buckets": [
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key": 961.0
                },
                {
                    "avg_field": {
                        "value": 3080.0
                    },
                    "doc_count": 1,
                    "key": 980.0
                },
                {
                    "avg_field": {
                        "value": 3077.0
                    },
                    "doc_count": 1,
                    "key": 1176.0
                },

So generally we can get 3 different results for one query.

PS: Elasticsearch compatible URL has the same behaviour

Expected behavior
It should return the dataset like provided below

{
    "num_hits": 1165439,
    "hits": [],
    "elapsed_time_micros": 6665,
    "errors": [],
    "aggregations": {
        "comment_ranking_avg": {
            "buckets": [
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key": 961.0
                },
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key":  928.0
                },
                {
                    "avg_field": {
                        "value": 3504.0
                    },
                    "doc_count": 1,
                    "key": 730.0
                },
..... 

Configuration:
Please provide:

  1. Output of quickwit --version
Quickwit 0.8.1 (aarch64-unknown-linux-gnu 2024-03-29T14:09:41Z e6c5396)
  1. The index_config.yaml
 Provided in the attached archive)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.