lance-format / lance-format/lance
Track removing Brotli from pylance's DataFusion FFI dependency graph
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
pylance currently keeps the Python DataFusion FFI integration exposed through FFILanceTableProvider. Keeping that API means pylance still pulls in Brotli through DataFusion's FFI dependency graph:
pylance
-> datafusion-ffi
-> datafusion-proto
-> default = ["parquet"]
-> datafusion-datasource-parquet
-> parquet
-> brotli
Downstream cannot disable this by adding datafusion-proto = { default-features = false } in pylance, because Cargo features are additive. The datafusion-ffi -> datafusion-proto dependency edge enables datafusion-proto/default, so pylance cannot turn it off from its own manifest.
This also exposes us to the current Brotli allocator resolution issue. A fresh Python lockfile resolve can select alloc-stdlib 0.2.3, which depends on alloc-no-stdlib >=2.0.4, <4.0.0, while brotli 8.0.3 also depends on alloc-no-stdlib 2.0. Cargo may resolve both alloc-no-stdlib 2.0.4 and 3.0.0, causing Brotli to fail to compile with allocator trait mismatches.
For now we should keep the Python FFI API stable and pin the Python-side allocator dependency resolution to the known working combination. Longer term, we should remove this pin once upstream DataFusion makes datafusion-ffi depend on datafusion-proto with default-features = false, or otherwise exposes a feature path that lets downstream users avoid Parquet/Brotli when they only need FFI table provider support.
Acceptance criteria:
pylancekeeps exportingFFILanceTableProvider.- Python CI continues to pass with
--locked. - The temporary allocator pin is removed once the upstream DataFusion dependency edge no longer forces
datafusion-proto/default. cargo tree --manifest-path python/Cargo.toml -i brotlino longer finds Brotli after the upstream dependency graph is fixed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with python/Cargo.toml and inspect the datafusion-ffi dependency path, then run cargo tree --manifest-path python/Cargo.toml -i brotli. Check the Python lockfile and Python CI with --locked; done means FFILanceTableProvider remains exported, CI passes, and Brotli is absent once upstream removes the forcing dependency edge.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system, ci-cd
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100