[DISCUSS] Consider Vendoring Certain Dependencies
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion has many dependencies that leads to increased binary size and compilation while saving us the trouble of maintaining those implementations in our own codebase.
### Describe the solution you'd like
This issue is to identify and collect any suggestions for crate/dependency `shallow` and `stable` enough to justify having an In-house implementation.
'shallow': We do not use much of the original crate. i.e we just use a small part that can be implemented in house without having to implement/port a massive dependency tree that crate has. The shallower the dependency, the more beneficial an in-house implementation would be.
'stable': The code is mostly stable and will not require much active changes after having an In-house implementation. (While all code eventually requires some changes, it is about how frequently and disruptively it will give us a headache.)
### Describe alternatives you've considered
Do not have any in-house implementation. Pay the price of increased binary size and compile time.
### Additional context
_Originally suggested by @ozankabak while discussing my GSoC 2025 Proposal for "Optimizing compile time and binary size" over discord and before that in #14478._
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing DataFusion's dependency choices and the discussion referenced in #14478, then identify crates that are both shallow and stable enough to replace in-house. Done means producing justified dependency candidates and deciding whether each replacement is worthwhile.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- build-system, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100