Moving the DataFusion integration into a separate repository
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
Continuing the https://github.com/apache/iceberg-rust/discussions/2992 discussion in an issue as requested by @andygrove, so that updates get sent to the mailing list.
Citing the original discussion description:
---
I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing [DataFusion integration](https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion) into a crate maintained in a separate repository.
As discussed in https://github.com/datafusion-contrib/datafusion-distributed/issues/599, we are exploring incorporating the integration into [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed), making native distributed Iceberg support a core offering of that project. Another option worth exploring would be to create a dedicated repository, such as datafusion-contrib/datafusion-iceberg.
Regardless of its eventual home, the goal would be to allow the community to advance the Iceberg+DataFusion integration without adding review pressure for iceberg-rust maintainers. The integration could evolve at its own pace and be maintained by a different or broader group of contributors.
Some reasons why this may be worth considering:
- There appears to be a vision of making iceberg-rust engine-agnostic (https://github.com/apache/iceberg-rust/pull/2613#issuecomment-4830211788, cc @blackmwk).
- Several integration-related PRs have had difficulty receiving review attention in this repository, such as https://github.com/apache/iceberg-rust/pull/2671 and https://github.com/apache/iceberg-rust/pull/2521.
- Other DataFusion-related projects have expressed interest in hosting an Iceberg integration, including https://github.com/apache/datafusion-ballista/issues/2174 and https://github.com/datafusion-contrib/datafusion-distributed/issues/599.
I’d like to hear from the community (particularly the iceberg-rust maintainers) about the possibility of moving iceberg-datafusion out of the iceberg-rust repository. This could allow projects such as [Ballista](https://github.com/apache/datafusion-ballista) and [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) to share common functionality and contribute to the integration independently of the iceberg-rust release and review lifecycle.
Contributor guide
Research direction
Start by reading the discussion at datafusion-contrib/datafusion-distributed#599 and the existing crates/integrations/datafusion directory, then review the linked Ballista issue and related pull requests. Done means the community and maintainers agree whether the integration should move, identify its destination, and define a migration plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100