apache / apache/datafusion-ballista

Support for custom `ParquetFileReaderFactory`

Open
#186 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)

DataFusion recently added a way to provide a user-defined `AsyncFileReader` to `ParquetExec`. Currently there is no way to leverage this in Ballista since the deserialization logic will construct a `ParquetExec` with the default implementation (which essentially just uses the registered `ObjectStore`).

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

We should be able to leverage this feature in Ballista without overriding the entire serialization logic for physical plans.

I see one of two approaches here:

1. Push this back into DataFusion and allow registration of custom `ParquetFileReaderFactory` in the `SessionContext` somewhere in which case it should be trivial to support in Ballista.
2. Add this capability in ballista explicitly.

For option 2, we might consider using the `PhysicalExtensionCodec` for this. We could add methods:

```
/// The deserialization logic will invoke this method for any `PhysicalPlanType::ParquetScan` nodes in the serialized plan. If a custom deserialization is required, apply it and return the deserialized result, otherwise return None and the deserialization will fallback to the default
fn try_decode_parquet_exec(
&self,
scan: &ParquetScanExecNode,
) -> Result, BallistaError> {
Ok(None)
}

```

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Ballista's deserialization of PhysicalPlanType::ParquetScan and the PhysicalExtensionCodec path, then compare how ParquetExec currently obtains its AsyncFileReader. The work is done when a custom ParquetFileReaderFactory can be supplied without overriding the entire physical-plan serialization logic, with the chosen SessionContext or extension-codec approach documented and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.