quickwit-oss / quickwit-oss/quickwit

Aggregation failing after index schema update

Open
#6,130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
We have had an index on which we defined a text field change with fast: false , this index already has data. We could not perform aggregation queries on it. When we attempted aggregation queries on it we got this error:

{ ERROR quickwit_search::error: search internal error: search failed for the following splits:  x, y, z,  . For instance, split x failed with the following error message: internal error: `An invalid argument was passed: 'Field \"change\" is not configured as fast field'`"}

We have now updated the schema to set fast: true so we could perform aggregation even if old split will be considered as missing data. however we still the same error after changing the schema and ingesting new data.

{ ERROR quickwit_search::error: search internal error: search failed for the following splits:  x, y, z,  . For instance, split x failed with the following error message: internal error: `An invalid argument was passed: 'Field \"change\" is not configured as fast field'`"}

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  1. create an index with a field test that's is not fast-field
  2. ingest some data
  3. change the schema to make the field test fast-fields
  4. ingest more data
  5. attempt to make aggregation query on the field test
  6. You should be able to observe the reported error

Expected behaviour
We expected after the schema update, that aggregation queries on change field should work although it will not take into account old split on which fast-field was not enabled on the change field.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue by creating an index with a non-fast test field, ingesting data, enabling fast fields, ingesting more data, and running an aggregation. Trace schema updates, ingestion, and aggregation over old and new splits; done means the aggregation succeeds while treating old splits as missing data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.