quickwit-oss / quickwit-oss/quickwit
Better resource utilisation for SQS File source
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Is your feature request related to a problem? Please describe.
Currently, the SQS file source exhibits a limited indexing rate when dealing with small files (few MB).
Describe the solution you'd like
Ideally, the indexing would be CPU bound like for other sources.
Describe alternatives you've considered
A workaround is currently to create more indexing pipelines than with other sources.
Additional context
Times measured performing the different async state changes in the coordinator
| event | total_time_spent |
|---|---|
| read_batch | 159 |
| truncate | 2 |
| start_processing | 164 |
| polling_messages | 13 |
It seems half the source time is spent opening the file objects.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the SQS file source workload with small files and profiling the coordinator's async state changes, especially file opening, read_batch, and start_processing. Compare the indexing rate with other sources and the timing table in the issue. Done means the source achieves materially better resource utilisation and approaches CPU-bound indexing without requiring extra pipelines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100