datafusion-contrib / datafusion-contrib/datafusion-distributed

[Iceberg] Add remote benchmarks

Open
#602 1 comment 0 reactions 1 assignee View on GitHub

@gabotechs is already working on this.

Since Aug 9, 2026.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

Part of #599.

Add repeatable remote benchmarks for the Iceberg integration.

Scope:

  • Extend the remote benchmark environment to register Iceberg tables from object storage.
  • Document required object-store configuration and benchmark invocation.
  • Run representative Iceberg workloads against a remotely accessible table.
  • Establish a baseline for comparing local and remote execution.

Ideally, the same datasets that are supported for raw parquet files should also be supported for Iceberg.

They probably will have a different shape, and as the remote benchmarks are designed to run on AWS, they will probably need to be wired up to AWS Glue or something.

As this requires an expensive infra setup, this should be handled by a Datadog employee first, even if the remote setup should work for everyone at the end of the day.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.