quickwit-oss / quickwit-oss/quickwit
Storage configuration for Azure or S3
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Is your feature request related to a problem? Please describe.
When deploying Quickwit via a Helm chart and using a storage backend like AWS or Azure, there's frustration due to unclear or undocumented schema for configuring default_index_root_uri and metastore_uri specifically for Azure.
Describe the solution you'd like
Following point should be enhanced imho
While configuring the storage backend for Azure, the schemas for default_index_root_uri and metastore_uri should be more clear and documented. Currently, the assumed format is:
default_index_root_uri: <s3|azure>://<bucket|container>/<path>
metastore_uri: <s3|azure>://<bucket|container>/<path>
This format might not be intuitive for users. An improved version could be:
default_index_root_uri: <s3|azure>://<storage_account>/<bucket|container>
metastore_uri: <s3|azure>:<s3|azure>://<storage_account>/<bucket|container>
Alternatively, for enhanced clarity and user experience, users could provide configuration as follows:
config:
# Storage configuration.
storage:
azure:
account: "mystorageaccount"
access_key: "somesecretkeys"
container: "quickwit-container"
metastore_uri: azure://{account}/{container} # Auto-generated based on configuration info from storage
default_index_root_uri: azure://{account}/{container} # Auto-generated
This approach ensures better clarity and ease of understanding for users configuring the Azure storage backend.
Creating manually indexes
Currently, users have to create indexes manually via curl on the storage, specifying the index name and schema. This is obviously an anti-pattern for automation. The initial index should be created during deployment, with an eventual configuration parameter.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing the Helm chart storage configuration and the URI handling for Azure and S3, then clarify which configuration format should be supported and whether automatic initial index creation belongs in this issue. Done means the selected storage schema is documented and the deployment-time index behavior is specified and tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, helm, rust
- Domain
- backend, cloud, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100