lance-format / lance-format/lance
Add out-of-batch coalescing
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
#1959 describes coalescing in general and should add it for IOPS within a batch. However, there are cases where it makes sense to coalesce requests between batches, especially on filesystems with a high "coalesce aggressiveness". There are two problems with out-of-batch coalescing.
First, we can't reasonably expect these IOPS to be ordered. We could keep an ordered queue of IOPS in the scheduler but, "file offset order" is not the same as "priority order" and so we'd have to maintain both queues and that would get potentially complicated and expensive.
A better approach might be to have a task that periodically runs on the "pending but not submitted" IOPS queue. It could clone the queue, reorder it, coalesce requests as appropriate, restore the original order and then merge this with the original queue (which might have popped off a few requests in the meantime). This could be quite complicated, especially since a high priority IOP might be coalesced with a low priority IOP.
Out of batch coalescing might thus only be enabled on slower filesystems.
The second concern is that we could potentially coalesce too much. Imagine we are reading an entire file. If our coalesce criteria is simply based on "coalesce aggressiveness" then we would end up coalescing the entire file read into a single IOP. We will need to avoid this by introducing some criteria / configuration that we can use to know when a request has grown big enough to stop trying to coalesce.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating the scheduler's pending-but-not-submitted IOPS queue and the existing in-batch coalescing described by #1959; clarify ordering, priority, filesystem gating, and maximum request size before implementation. Done means out-of-batch coalescing has defined limits and behavior for these cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100