apache / apache/datafusion

Add support to write DataFrames to an object store

Open
#3,668 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Hey folks :wave: we're evaluating Datafusion/Ballista to perform queries as part of a pipeline on some pretty large datasets (we're talking on the order of a few Terabytes at minimum) and would like to read from Parquet files from GCS and write as Parquet to a different GCS bucket. Right now it's possible to get a DataFrame from GCS using the object_store crate, but it's not possible to write the resulting DataFrame back to GCS.

**Additional context**
This feature was already discussed in #2185 but that issue is out of date because the new object_store crate has since been merged in.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing object_store crate integration for reading Parquet files from GCS, then trace how a DataFrame is currently written to Parquet. Define the expected behavior for writing the resulting DataFrame to a different GCS bucket and verify it with an end-to-end object-store case.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.