lance-format / lance-format/lance
feature: add configurable fail-fast scheduler error handling
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Summary
Add a session-level configuration option that controls how the Lance scan
scheduler handles an error from one physical I/O task within a logical read
request.
Proposed modes:
best_effort— preserve the existing behavior and remain the default.fail_fast— return the first observed error and cancel the remaining
physical tasks belonging to the same logical request.
Current behavior and drawbacks
A logical Lance read may be split into multiple physical range reads. If one
range read fails, the overall logical request cannot produce a valid result.
However, the current scheduler behavior may allow the remaining reads to
continue.
In the standard scheduler, the error is not returned until all sibling tasks
complete. The lite scheduler processes task handles in request order, so an
error from a later task may not be noticed while an earlier task remains
blocked.
This has several drawbacks:
- Error reporting can be delayed by slow, blocked, or retrying sibling reads.
- Storage, network, CPU, and memory resources continue to be consumed for a
request that can no longer succeed. - Queued tasks occupy scheduler capacity and create backpressure for unrelated
requests. - During a storage incident, unnecessary reads and retries can amplify load on
an already degraded service. - Interactive engines such as Spark SQL may remain blocked longer than
necessary before reporting the actual storage error. - The timing of error propagation differs between the standard and lite
scheduler implementations.
The existing behavior does not provide partial recovery: the logical request
still eventually fails.
Why this should be configurable
Changing the default behavior could affect existing workloads and cancellation
timing. Keeping best_effort as the default preserves compatibility, while
allowing latency-sensitive or resource-constrained deployments to opt into
fail_fast.
This is naturally a session-level operational policy. Query engines generally
construct Lance datasets and schedulers internally, so end users do not have
direct access to scheduler APIs. Engines should be able to configure the policy
once for a session and propagate it through Lance storage options whenever a
table is opened.
For example, a Spark session using a Lance catalog could set:
spark-sql \
--conf "spark.sql.catalog.lance.storage.scheduler_error_mode=fail_fast"
A Python-based engine could similarly add the setting to the storage options
used for every dataset opened during the session:
session_storage_options = {
"scheduler_error_mode": "fail_fast",
}
dataset = lance.dataset(uri, storage_options=session_storage_options)
Using storage options avoids adding per-scan configuration APIs and follows the
existing engine-to-Lance configuration path.
Proposed semantics
Add a scheduler_error_mode storage option with the following values:
best_effort: preserve the existing backend-specific behavior.fail_fast: when a physical task reports an error:- Preserve and return the original error.
- Cancel queued and, where supported, in-flight sibling tasks from the same
logical scheduler request. - Do not cancel tasks belonging to unrelated requests.
- Keep scheduler backpressure and I/O accounting consistent after
cancellation.
The option should default to best_effort, and invalid values should produce a
descriptive configuration error.
Configured object-store retry policies should remain unchanged. Fail-fast
handling begins only after a physical task ultimately reports an error to the
scheduler.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the standard and lite scheduler implementations and the storage-options configuration path described in the issue. Trace how physical task errors and logical requests are handled, then identify the scheduler and configuration tests that cover both modes; done means preserving best_effort behavior while fail_fast cancels only sibling tasks and reports invalid values clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, spark
- Domain
- backend, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100