apache / apache/iceberg-rust

Moving the DataFusion integration into a separate repository

Open
#3,029 36 comments 9 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

Continuing the https://github.com/apache/iceberg-rust/discussions/2992 discussion in an issue as requested by @andygrove, so that updates get sent to the mailing list.

Citing the original discussion description:

---

I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing [DataFusion integration](https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion) into a crate maintained in a separate repository.

As discussed in https://github.com/datafusion-contrib/datafusion-distributed/issues/599, we are exploring incorporating the integration into [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed), making native distributed Iceberg support a core offering of that project. Another option worth exploring would be to create a dedicated repository, such as datafusion-contrib/datafusion-iceberg.

Regardless of its eventual home, the goal would be to allow the community to advance the Iceberg+DataFusion integration without adding review pressure for iceberg-rust maintainers. The integration could evolve at its own pace and be maintained by a different or broader group of contributors.

Some reasons why this may be worth considering:

- There appears to be a vision of making iceberg-rust engine-agnostic (https://github.com/apache/iceberg-rust/pull/2613#issuecomment-4830211788, cc @blackmwk).
- Several integration-related PRs have had difficulty receiving review attention in this repository, such as https://github.com/apache/iceberg-rust/pull/2671 and https://github.com/apache/iceberg-rust/pull/2521.
- Other DataFusion-related projects have expressed interest in hosting an Iceberg integration, including https://github.com/apache/datafusion-ballista/issues/2174 and https://github.com/datafusion-contrib/datafusion-distributed/issues/599.

I’d like to hear from the community (particularly the iceberg-rust maintainers) about the possibility of moving iceberg-datafusion out of the iceberg-rust repository. This could allow projects such as [Ballista](https://github.com/apache/datafusion-ballista) and [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) to share common functionality and contribute to the integration independently of the iceberg-rust release and review lifecycle.

Contributor guide

Open the contributing guide

Research direction

Start by reading the discussion at datafusion-contrib/datafusion-distributed#599 and the existing crates/integrations/datafusion directory, then review the linked Ballista issue and related pull requests. Done means the community and maintainers agree whether the integration should move, identify its destination, and define a migration plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.