feature: unified FileSystem abstraction over local and object storage

RFC: Unified FileSystem abstraction over local and object storage

Status: Proposal
Area: lance-io
Related: file-object-store:// scheme, WrappingObjectStore, IOTracker

Problem

Lance's I/O layer hardcodes the choice between local and object-storage access
in several places. ObjectStore::open, open_with_size, and create branch
on the URI scheme at runtime:

// lance-io/src/object_store.rs:847
pub async fn open(&self, path: &Path) -> Result<Box<dyn Reader>> {
    match self.scheme.as_str() {
        "file" if self.has_direct_local_paths() => {
            LocalObjectReader::open_with_tracker(...)   // direct std::fs
        }
        "file+uring" => { UringReader::open(...) }      // Linux io_uring
        _ => { CloudObjectReader::new(...) }            // object store
    }
}

The same match self.scheme / has_direct_local_paths() pattern is repeated
in at least six places (object_store.rs:848, 910, 973, 1230, 1433, 1509),
covering open, open_with_size, create, copy_via_stream, copy_impl,
and remove_dir_all.

Why this is a problem

1. file-object-store:// is a workaround for this inconsistency. The
   file scheme deliberately bypasses the ObjectStore trait for performance
   (LocalObjectReader / LocalWriter go straight to std::fs). This means
   WrappingObjectStore wrappers (e.g. IOTrackingStore, FailingProxyStore,
   ThrottledStoreWrapper) cannot intercept local I/O. To work around this,
   Lance ships a second scheme, file-object-store://, that routes local I/O
   through the ObjectStore API. See the comment in
   lance-io/src/object_store/providers.rs:358-367:

   // The "file" scheme has special optimized code paths that bypass
   // the ObjectStore API for better performance. However, this can make it
   // hard to test when using ObjectStore wrappers, such as IOTrackingStore.
   // So we provide a "file-object-store" scheme that uses the ObjectStore API.
   

   The cost: users must use file-object-store:// to exercise wrappers, but
   production uses file://, so test and production behavior diverge. The
   scheme also leaks into special cases throughout object_store.rs
   (:1527, :1567).

2. Fault injection and observability cannot be unified. IOTracker can
   observe local I/O (via record_read / record_write), but cannot
   inject faults into it. Injecting latency or errors into local reads and
   writes requires the file-object-store:// workaround. There is no single
   interception point that covers both local and cloud I/O.

3. Adding a backend means editing every branch. Each new local
   acceleration or cloud backend must touch all six match self.scheme
   sites, and the open / open_with_size branches are near-duplicates.

Proposal

Introduce a unified FileSystem trait (the Lance analogue of DuckDB's
FileSystem interface) that abstracts "open a file for read/write" over both
local disk and object storage. The concrete implementation is chosen at
construction time, not by a runtime match on the scheme string.

// lance-io/src/file_system.rs (new)
#[async_trait]
pub trait FileSystem: Send + Sync + Debug {
    async fn open(&self, path: &Path) -> Result<Box<dyn Reader>>;
    async fn open_with_size(&self, path: &Path, known_size: usize) -> Result<Box<dyn Reader>>;
    async fn create(&self, path: &Path) -> Result<Box<dyn Writer>>;
    async fn read_range(&self, path: &Path, range: Range<usize>) -> Result<Bytes>;
    async fn read_all(&self, path: &Path) -> Result<Bytes>;
    async fn size(&self, path: &Path) -> Result<u64>;
    fn is_local(&self) -> bool;
}

Two implementations:

• LocalFileSystem — wraps the existing LocalObjectReader /
  LocalWriter (direct std::fs), preserving the performance optimization.
• CloudFileSystem — wraps the existing CloudObjectReader /
  ObjectWriter / SmallReader through the ObjectStore trait.

ObjectStore holds an Arc<dyn FileSystem> chosen in its constructor, and
open / open_with_size / create delegate to it instead of branching on
self.scheme.

Fault injection becomes uniform

A FaultInjectingFileSystem wrapper can intercept both local and cloud
I/O through one FileSystem implementation, eliminating the need for
file-object-store://:

#[async_trait]
impl FileSystem for FaultInjectingFileSystem {
    async fn open(&self, path: &Path) -> Result<Box<dyn Reader>> {
        self.injector.before("open", path)?;   // inject delay / error
        self.inner.open(path).await
    }
    // ...
}

Scope

This is a progressive change. The first step covers the core read/write
path only:

• New FileSystem trait + LocalFileSystem + CloudFileSystem (pure
  additions, no behavior change).
• ObjectStore holds a FileSystem and delegates open / open_with_size /
  create to it.
• The local optimizations in copy_via_stream, copy_impl, and
  remove_dir_all are left on is_local() for now, migrated in a follow-up.

Performance

The local path keeps its direct std::fs access — LocalFileSystem still
returns LocalObjectReader / LocalWriter. The only change is that the
runtime match self.scheme becomes a construction-time choice plus one
virtual dispatch. No measurable overhead is expected; this must be verified
with a benchmark before merge.

Risks and mitigations

Risk	Mitigation
Touches core I/O layer	Progressive: pure-additive first PR, then migration, then wrapper
Local performance regression	LocalFileSystem keeps direct std::fs; benchmark before merge
Test coverage (file / file-object-store / uring)	Keep existing #[case("file")] / #[case("file-object-store")] tests; add uring coverage
ObjectStore::local() / memory() direct constructors	Adapt them to build the matching FileSystem

Open questions

1. Should FileSystem also absorb copy and remove_dir_all, or keep those
   on ObjectStore with an is_local() fast path?
2. Should file-object-store:// be deprecated once the abstraction lands, or
   kept for backward compatibility?
3. Does the Reader / Writer handle abstraction already cover enough, such
   that we only need to abstract the selection logic rather than a full
   FileSystem trait?

Alternatives considered

• Factory function that centralizes the match self.scheme into one
  helper. Lower effort (~100-200 lines) but does not remove the hardcoding or
  enable uniform fault injection.
• Route local I/O through ObjectStore (drop the bypass). Simplest, but
  loses the local performance optimization that motivated the bypass.
• Status quo + file-object-store://. Zero code change, but keeps the
  workaround and the test/production divergence.