quickwit-oss / quickwit-oss/quickwit

Add a bench for QuickwitCollector and optimize it.

Open
#3,978 0 comments 0 reactions 2 assignees View on GitHub

@dayaffe is already working on this.

Since Aug 28, 2026.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

There are a lot of performance headroom for QuickwitCollector. This is our hottest code, we should have benchmarks to help its iterating on it.

Ideally we probably want to bench over >1million worth of documents from gharchive. I think we can use the schema from

version: 0.6
index_id: gh-archive
doc_mapping:
  field_mappings:
      - name: payload
        type: object
        field_mappings:
        - name: title
          type: text
        - name: body
          type: text
      - name: issue
        type: object
        field_mappings:
        - name: title
          type: text
        - name: body
          type: text
    - name: created_at
      type: datetime
      fast: true
      input_formats:
      - rfc3339
  timestamp_field: created_at
  mode: dynamic

This is probably something we don't want on the repo however.
Maybe we can have the data gzipped somewhere on the internet (s3? github?), and have a program that downloads the corpus and builds the split. (let's not make it part of the workspace)

Benchmarking at the tantivy level is ok.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.