datafusion-contrib / datafusion-contrib/datafusion-distributed

[EPIC] Iceberg integration

Open

#599 opened on Aug 9, 2026 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language: Rust
Stars: 139
Forks: 67
Avg merge: 3d 1h
Merged PRs (30d): 35

Description

This is a collection of issues and central discussion for building a native integration to Iceberg inside this project.

The reason why having a native integration here is because:

Serve as a testing use-case for this project. During CI, tests against real Iceberg tables using distributed execution will run, increasing coverage of the core of datafusion-distributed itself.
Guide design decisions regarding public APIs. Right now, the only real use-case present in this project is raw parquet tables stored in a file system, but that use-case is relatively simple and misses on features like dynamically resolved files during execution. Having a more complex use-case like Iceberg will help shape the project's APIs.
Lack of basic functionality in upstream integration https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion, not enough review power there (e.g., https://github.com/apache/iceberg-rust/pull/2298, https://github.com/apache/iceberg-rust/pull/2521, https://github.com/apache/iceberg-rust/issues/2364), and maintainers explicitly stating their willingness to not maintain a DataFusion integration there long term (e.g., https://github.com/apache/iceberg-rust/pull/2613#issuecomment-4830211788)

The extension points in this project seem to fit very well for providing a good distributed integration with DataFusion and Iceberg, but the task is big and will need to be broken down in several sub issues that people can contribute to independently:

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start by reviewing the linked sub-issues 600–616 and the referenced Apache Iceberg DataFusion integration discussions. The work is complete only when the independently scoped sub-issues collectively provide the planned native Iceberg integration and distributed CI coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack: rust
Domain: databases, distributed-systems
Issue type: Feature
Difficulty: 5/5
Estimated time: Over a week
Activity status: Active
Clarity: Needs clarification
Newbie friendliness: 25/100