apache / apache/datafusion

Reduce Recompilation When Switching Crates

Open
#7,370 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

It is fairly common to compile the various DataFusion crates separately for the purposes of improved test iteration. Unfortunately the experience of switching between crates is currently often involves recompiling huge tranches of the build tree.

This is because different dependencies set different feature flags for various dependencies, in some cases these are very low-level crates, forcing recompilation of the entire dependency tree. This is not only slow, but also is a recipe for ending up with an extremely chonky target directory

Examples of particularly painful crates with inconsistent feature sets:

* syn
* chrono
* num-traits
* tokio
* regex
* hashbrown

### Describe the solution you'd like

One possible solution is to create a workspace hack crate that forces a common set of feature flags when building crates within the workspace. There is [tooling](https://docs.rs/cargo-hakari/0.9.27/cargo_hakari/) to make this slightly less painful.

The major downsides of this are:

* Might complicate consuming DataFusion as a git dependency
* Would require a slightly different release workflow `cargo hakari publish` instead of `cargo publish`

A slightly less intrusive solution might be to try and find particularly problematic dependencies and make them workspace dependencies

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the workspace dependency feature sets for syn, chrono, num-traits, tokio, regex, and hashbrown. Compare a cargo-hakari workspace hack with making selected dependencies workspace dependencies, then verify that switching between DataFusion crates avoids broad recompilation without breaking git consumption or the release workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.