quickwit-oss / quickwit-oss/quickwit

Storage configuration for Azure or S3

Open
#4,797 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Is your feature request related to a problem? Please describe.
When deploying Quickwit via a Helm chart and using a storage backend like AWS or Azure, there's frustration due to unclear or undocumented schema for configuring default_index_root_uri and metastore_uri specifically for Azure.

Describe the solution you'd like

Following point should be enhanced imho

While configuring the storage backend for Azure, the schemas for default_index_root_uri and metastore_uri should be more clear and documented. Currently, the assumed format is:

default_index_root_uri: <s3|azure>://<bucket|container>/<path>
metastore_uri: <s3|azure>://<bucket|container>/<path>

This format might not be intuitive for users. An improved version could be:

default_index_root_uri: <s3|azure>://<storage_account>/<bucket|container>
metastore_uri: <s3|azure>:<s3|azure>://<storage_account>/<bucket|container>

Alternatively, for enhanced clarity and user experience, users could provide configuration as follows:

config:
  # Storage configuration.
  storage:
    azure:
      account: "mystorageaccount"
      access_key: "somesecretkeys"
      container: "quickwit-container"
  metastore_uri: azure://{account}/{container} # Auto-generated based on configuration info from storage
  default_index_root_uri: azure://{account}/{container} # Auto-generated

This approach ensures better clarity and ease of understanding for users configuring the Azure storage backend.

Creating manually indexes

Currently, users have to create indexes manually via curl on the storage, specifying the index name and schema. This is obviously an anti-pattern for automation. The initial index should be created during deployment, with an eventual configuration parameter.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by tracing the Helm chart storage configuration and the URI handling for Azure and S3, then clarify which configuration format should be supported and whether automatic initial index creation belongs in this issue. Done means the selected storage schema is documented and the deployment-time index behavior is specified and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure, helm, rust
Domain
backend, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.