datafusion-contrib / datafusion-contrib/datafusion-distributed
[Iceberg] Add remote benchmarks
@gabotechs is already working on this.
Since Aug 9, 2026.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Part of #599.
Add repeatable remote benchmarks for the Iceberg integration.
Scope:
- Extend the remote benchmark environment to register Iceberg tables from object storage.
- Document required object-store configuration and benchmark invocation.
- Run representative Iceberg workloads against a remotely accessible table.
- Establish a baseline for comparing local and remote execution.
Ideally, the same datasets that are supported for raw parquet files should also be supported for Iceberg.
They probably will have a different shape, and as the remote benchmarks are designed to run on AWS, they will probably need to be wired up to AWS Glue or something.
As this requires an expensive infra setup, this should be handled by a Datadog employee first, even if the remote setup should work for everyone at the end of the day.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.