lance-format / lance-format/lance
Process-wide concurrency limit should be scoped to object stores instead
Open
@Xuanwo is already working on this.
Since Mar 9, 2026.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently we have a process-wide concurrency limit (defaults to 128) in addition to the per-scan concurrency limit. Now that we have object store caching (so we reuse the same object store instance) and we have scenarios with multiple buckets we should change it so there is one single concurrency limit which is enforced at the object store level. Then we can pull the concurrency limits out of the scan scheduler.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.