quickwit-oss / quickwit-oss/quickwit
Add native support for Amazon Elastic File System (EFS)
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
After the changes that have been made, distributed search is not available on file system. This also means the tutorial on the website is broken (Tutorial Link).
Error Message
❯ ./quickwit run --service searcher --config ./searcher-1.yaml
2022-05-24T10:51:00.280Z INFO quickwit: version="0.2.1" commit="e426a45"
2022-05-24T10:51:00.280Z WARN quickwit_config::config: Cluster ID is not set, falling back to default value. cluster_id="quickwit-default-cluster"
2022-05-24T10:51:00.280Z INFO quickwit_cli: Loaded Quickwit config. config_uri=file:///home/aimless/quickwit_test/quickwit/target/release/searcher-1.yaml config=QuickwitConfig { version: 0, node_id: "searcher-1", cluster_id: "quickwit-default-cluster", listen_address: "127.0.0.1", rest_listen_port: 7280, gossip_listen_port: 7280, grpc_listen_port: 7281, peer_seeds: ["127.0.0.1:7290", "127.0.0.1:7300"], data_dir_path: "./qwdata", metastore_uri: Uri { uri: "file:///home/aimless/quickwit_test/quickwit/target/release/qwdata/indexes#polling_interval=30s", protocol_idx: 4 }, default_index_root_uri: Uri { uri: "file:///home/aimless/quickwit_test/quickwit/target/release/qwdata/indexes", protocol_idx: 4 }, indexer_config: IndexerConfig { split_store_max_num_bytes: Byte(100000000000), split_store_max_num_splits: 1000 }, searcher_config: SearcherConfig { fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), max_num_concurrent_split_streams: 100 }, storage_config: None }
Command failed: Cluster feature cannot be used in conjunction with a file backed metastore.
It would make sense to make distributed search possible on file system as we can use AWS EFS for that.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the shown quickwit run --service searcher --config ./searcher-1.yaml command and inspect the configuration that triggers the file-backed metastore restriction. Done means distributed search works with a file system backed by EFS and the linked HDFS logs tutorial is no longer broken.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100