apache / apache/datafusion

Consider providing an option for dynamic linking

Open
#6,470 5 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`datafusion` is a "swiss army knife" kind of library - it's large and developing against it means **~13 second link times** every time you re-run tests, even with modern linkers like `mold`.

I recently tried the approach described in [this blog post](https://robert.kra.hn/posts/2022-09-09-speeding-up-incremental-rust-compilation-with-dylibs/) - wrapping `datafusion` into a dynamic library. This reduced my incremental test build times by ~10 seconds!

### Describe the solution you'd like

Bevy project (also a very large library) now [offer a feature flag](https://bevyengine.org/learn/book/getting-started/setup/#enable-fast-compiles-optional) to link it dynamically.

Specifying `crate-type = ["rlib", "dylib"]` might also be enough to let users easily link `datafusion` dynamically in their `dev` builds using `RUSTFLAGS="-C prefer-dynamic"`.

It may iterations speeds for `datafusion`'s own development.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reviewing the repository's Cargo manifests and development build configuration, then try crate-type = ["rlib", "dylib"] with RUSTFLAGS="-C prefer-dynamic" for DataFusion and its tests; done means dynamic linking is available without breaking normal builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, developer-experience
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.