lance-format / lance-format/lance

Add out-of-batch coalescing

Open
#1,960 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

#1959 describes coalescing in general and should add it for IOPS within a batch. However, there are cases where it makes sense to coalesce requests between batches, especially on filesystems with a high "coalesce aggressiveness". There are two problems with out-of-batch coalescing.

First, we can't reasonably expect these IOPS to be ordered. We could keep an ordered queue of IOPS in the scheduler but, "file offset order" is not the same as "priority order" and so we'd have to maintain both queues and that would get potentially complicated and expensive.

A better approach might be to have a task that periodically runs on the "pending but not submitted" IOPS queue. It could clone the queue, reorder it, coalesce requests as appropriate, restore the original order and then merge this with the original queue (which might have popped off a few requests in the meantime). This could be quite complicated, especially since a high priority IOP might be coalesced with a low priority IOP.

Out of batch coalescing might thus only be enabled on slower filesystems.

The second concern is that we could potentially coalesce too much. Imagine we are reading an entire file. If our coalesce criteria is simply based on "coalesce aggressiveness" then we would end up coalescing the entire file read into a single IOP. We will need to avoid this by introducing some criteria / configuration that we can use to know when a request has grown big enough to stop trying to coalesce.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating the scheduler's pending-but-not-submitted IOPS queue and the existing in-batch coalescing described by #1959; clarify ordering, priority, filesystem gating, and maximum request size before implementation. Done means out-of-batch coalescing has defined limits and behavior for these cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.