Proposal: add a Foyer-backed cache for immutable Lance data files

Motivation

Repeated scans of datasets stored in remote object storage may read the same Lance data ranges multiple times.

Lance's existing Session caches already cover metadata and parsed index structures, but they do not cache data-file contents.

Proposal

Add an optional cache in lance-c backed by Foyer.

The initial implementation would:

• Cache raw range-read bytes from immutable data/*.lance files.
• Support memory and disk caching through Foyer.
• Keep the cache disabled by default and make its capacity and local path configurable.
• Bypass the cache for manifests, transactions, deletion files, index files, and other objects.
• Keep the existing Lance Session metadata and index caches unchanged.

Proposed architecture

The cache would be inserted into the object-store range-read path:

flowchart TD
    A[Lance scanner / reader] --> B[ObjectStore cache wrapper]
    B --> C{Path matches data/*.lance?}

    C -->|No| D[Underlying ObjectStore]
    D --> E[Return range bytes]

    C -->|Yes| F[Foyer cache]
    F -->|Cache hit| E
    F -->|Cache miss| G[Underlying ObjectStore]
    G --> H[Populate Foyer]
    H --> E

Only immutable Lance data files enter Foyer. All other objects bypass it and continue using Lance's existing read path.

Cache responsibilities

flowchart LR
    A[Lance Dataset] --> B[Session metadata cache]
    A --> C[Session index cache]
    A --> D[Foyer data cache]

    B --> B1[Manifest and transaction metadata]
    B --> B2[Decoded deletion vectors]
    B --> B3[File metadata]

    C --> C1[Opened indices]
    C --> C2[Posting lists and positions]
    C --> C3[Index pages and structures]

    D --> D1[Raw range bytes]
    D1 --> D2[data/*.lance]

This keeps the responsibilities separate:

• Session metadata cache stores parsed dataset metadata.
• Session index cache stores parsed and opened index structures.
• Foyer stores raw range-read bytes from immutable Lance data files.

Feedback is welcome on the proposed scope, configuration API, and whether Foyer is an appropriate dependency for lance-c.

Start by tracing the object-store range-read path used by the Lance scanner/reader and reviewing how the existing Session metadata and index caches are separated. Evaluate the proposed Foyer-backed cache for immutable data/*.lance range bytes, configurable memory and disk capacity, and bypass behavior for other objects; done means the scope and configuration API are settled and the cache responsibilities remain separate.