apache / apache/datafusion

[DISCUSS] Consider Vendoring Certain Dependencies

Open
#15,360 7 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion has many dependencies that leads to increased binary size and compilation while saving us the trouble of maintaining those implementations in our own codebase.

### Describe the solution you'd like

This issue is to identify and collect any suggestions for crate/dependency `shallow` and `stable` enough to justify having an In-house implementation.
'shallow': We do not use much of the original crate. i.e we just use a small part that can be implemented in house without having to implement/port a massive dependency tree that crate has. The shallower the dependency, the more beneficial an in-house implementation would be.

'stable': The code is mostly stable and will not require much active changes after having an In-house implementation. (While all code eventually requires some changes, it is about how frequently and disruptively it will give us a headache.)

### Describe alternatives you've considered

Do not have any in-house implementation. Pay the price of increased binary size and compile time.

### Additional context

_Originally suggested by @ozankabak while discussing my GSoC 2025 Proposal for "Optimizing compile time and binary size" over discord and before that in #14478._

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing DataFusion's dependency choices and the discussion referenced in #14478, then identify crates that are both shallow and stable enough to replace in-house. Done means producing justified dependency candidates and deciding whether each replacement is worthwhile.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.