lance-format / lance-format/lance

feature: add configurable fail-fast scheduler error handling

Open
#8,781 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Summary

Add a session-level configuration option that controls how the Lance scan
scheduler handles an error from one physical I/O task within a logical read
request.

Proposed modes:

  • best_effort — preserve the existing behavior and remain the default.
  • fail_fast — return the first observed error and cancel the remaining
    physical tasks belonging to the same logical request.

Current behavior and drawbacks

A logical Lance read may be split into multiple physical range reads. If one
range read fails, the overall logical request cannot produce a valid result.
However, the current scheduler behavior may allow the remaining reads to
continue.

In the standard scheduler, the error is not returned until all sibling tasks
complete. The lite scheduler processes task handles in request order, so an
error from a later task may not be noticed while an earlier task remains
blocked.

This has several drawbacks:

  • Error reporting can be delayed by slow, blocked, or retrying sibling reads.
  • Storage, network, CPU, and memory resources continue to be consumed for a
    request that can no longer succeed.
  • Queued tasks occupy scheduler capacity and create backpressure for unrelated
    requests.
  • During a storage incident, unnecessary reads and retries can amplify load on
    an already degraded service.
  • Interactive engines such as Spark SQL may remain blocked longer than
    necessary before reporting the actual storage error.
  • The timing of error propagation differs between the standard and lite
    scheduler implementations.

The existing behavior does not provide partial recovery: the logical request
still eventually fails.

Why this should be configurable

Changing the default behavior could affect existing workloads and cancellation
timing. Keeping best_effort as the default preserves compatibility, while
allowing latency-sensitive or resource-constrained deployments to opt into
fail_fast.

This is naturally a session-level operational policy. Query engines generally
construct Lance datasets and schedulers internally, so end users do not have
direct access to scheduler APIs. Engines should be able to configure the policy
once for a session and propagate it through Lance storage options whenever a
table is opened.

For example, a Spark session using a Lance catalog could set:

spark-sql \
  --conf "spark.sql.catalog.lance.storage.scheduler_error_mode=fail_fast"

A Python-based engine could similarly add the setting to the storage options
used for every dataset opened during the session:

session_storage_options = {
    "scheduler_error_mode": "fail_fast",
}

dataset = lance.dataset(uri, storage_options=session_storage_options)

Using storage options avoids adding per-scan configuration APIs and follows the
existing engine-to-Lance configuration path.

Proposed semantics

Add a scheduler_error_mode storage option with the following values:

  • best_effort: preserve the existing backend-specific behavior.
  • fail_fast: when a physical task reports an error:
    • Preserve and return the original error.
    • Cancel queued and, where supported, in-flight sibling tasks from the same
      logical scheduler request.
    • Do not cancel tasks belonging to unrelated requests.
    • Keep scheduler backpressure and I/O accounting consistent after
      cancellation.

The option should default to best_effort, and invalid values should produce a
descriptive configuration error.

Configured object-store retry policies should remain unchanged. Fail-fast
handling begins only after a physical task ultimately reports an error to the
scheduler.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the standard and lite scheduler implementations and the storage-options configuration path described in the issue. Trace how physical task errors and logical requests are handled, then identify the scheduler and configuration tests that cover both modes; done means preserving best_effort behavior while fail_fast cancels only sibling tasks and reports invalid values clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, spark
Domain
backend, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.