Fully encapsulate object I/O behind Lance ObjectStore; deprecate .inner

Summary

Make lance_io::object_store::ObjectStore the complete I/O boundary for dataset reads and writes. Upstream object_store::ObjectStore (exposed today as the public ObjectStore::inner field) becomes an implementation detail behind a Lance-owned store, with wrappers able to intercept both store-level operations and opened reader/writer operations.

Full design: design-object-store-encapsulation.md (in-repo working doc).

Problem

Lance has two I/O layers. The Lance ObjectStore owns config/schemes/retry/parallelism, but inner is where user wrappers attach. Local fast paths bypass inner:

• open / open_with_size use LocalObjectReader / UringReader directly for file and file+uring.
• create uses LocalWriter directly for file.
• Local copy / remove_dir_all call filesystem helpers directly.

So a user object_store wrapper cannot reliably observe all I/O. IO statistics work around this with bespoke local record_read/record_write calls; throttling, fault injection, tracing, and auth have no clean hook.

Goal

Dataset / index / table code
        -> lance_io::object_store::ObjectStore
              -> Lance Reader / Writer wrappers
              -> backend (upstream object_store | std::fs::File | io_uring | memory | cloud)

All Lance I/O goes through Lance-owned methods (open, create, copy, delete, list, head, ...). Wrappers can decorate the store, returned readers, and returned writers. .inner becomes a deprecated compatibility escape hatch.

Success criteria

• No Lance read/write path accesses local files outside a Lance-level wrapper boundary.
• A wrapper can count exact local + cloud range reads through one mechanism.
• A wrapper can inject failures into local + cloud reads/writes without file-object-store://.
• Internal .inner uses are rare, documented, explicit compatibility escapes.
• Local optimized readers/writers remain available behind the new boundary.

Vertical slices

• Slice 1 (HITL): Lance-level wrapper surface + read-path tracer bullet — #7332
• Slice 2: Write-path through wrapped Lance Writer — #7333
• Slice 3: Reimplement IO stats as a pure store/reader/writer wrapper; delete local record_read/record_write special-cases — #7336
• Slice 4: Fault-injection / instrumentation parity test harness — #7337
• Slice 5: Migrate read .inner call sites — #7334
• Slice 6: Migrate mutation .inner call sites — #7335
• Slice 7: Deprecate .inner + explicit as_upstream_object_store() escape hatch — #7338

.inner call-site tracker

Migrate these to Lance ObjectStore methods. Checked off as each module is converted (Slices 5 & 6).

Read sites

• inner.get (~18) — lance/io/commit.rs:89, dataset/mem_wal/manifest.rs:112/308/322, dataset/mem_wal/wal.rs:1050, dataset.rs:1060, lance-io tests
• inner.head (~13) — lance-table/io/manifest.rs:42, lance-table/io/commit/external_manifest.rs:342, dataset/mem_wal/*, lance/index.rs:3142
• inner.list (~8) — dataset/mem_wal/wal.rs:1320/1348
• inner.list_with_delimiter (~8) — internal to object_store.rs
• inner.get_range (1) — lance-table/io/manifest.rs:50
• inner.get_opts (~7) — lance-io/object_reader.rs, object_store/list_retry.rs
• inner.get_ranges (~7) — object_reader.rs, list_retry.rs, throttle.rs

Mutation sites

• inner.delete (~16) — dataset/mem_wal/wal.rs, lance-namespace-impls/dir.rs:2145/2951, lance-table/io/commit.rs:1329, lance-io tests
• inner.copy (~6) — object_store.rs:829, tests
• inner.delete_stream (~7) — object_store.rs:878, list_retry.rs, throttle.rs
• inner.put_opts (~7) — object_writer.rs, list_retry.rs, throttle.rs
• inner.rename (1) — lance-table/io/commit.rs:191

Some lance-io-internal sites (list_retry.rs, throttle.rs, object_reader.rs, object_writer.rs) are the wrapper plumbing itself and may legitimately remain on the backend — to be classified during Slices 5/6.

Open questions (resolve in Slice 1)

1. Should WrappingLanceObjectStore replace WrappingObjectStore, or coexist long-term?
2. Local file opening: route through upstream get_opts + GetResultPayload::File, or keep opening std::fs::File directly behind a wrapped reader?
3. Post-deprecation .inner: remove, private with as_upstream_object_store, or compatibility module?
4. Represent copy/rename as reader/writer byte ops, or store-level metadata ops when the backend supports server-side copy?
5. Should throttling apply to opened-reader range calls, store-level requests, or both?

Deferred

• Phase 6 (privatize/remove .inner) — next breaking API window; not a slice yet.

Start with design-object-store-encapsulation.md, then inspect lance-io/object_store.rs and the listed .inner call sites. Use linked slices #7332–#7338 to choose an independently actionable change and follow that slice's tests. The umbrella work is done when its success criteria and all planned slices are complete.