apache / apache/datafusion

Project Hive Partition Columns With ProjectionExec

Open
#2,291 0 comments 1 reaction 0 assignees View on GitHub
enhancement help wanted
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Part of #2079

Currently the values of Hive partitions are projected within each of the various file format specific physical operators. As described in #2079 this has a number of drawbacks

**Describe the solution you'd like**

Rather than handling partition projection within the file scan operator, I would like to propose modifying `ListingTable` to add a `ProjectionExec` within `TableProvider::scan` instead of relying on the `FileFormat` implementations to do this. This `ProjectionExec` would be created with a set of literal expressions corresponding to the partition values

Therefore instead of `TableProvider::scan` generating something like

```
AvroExec:
```

It would generate

```
UnionExec
ProjectionExec: ... // Partition 1
AvroExec: ...
ProjectionExec: ... // Partition 2
AvroExec: ...
```

Note: this will depend on #2289

**Describe alternatives you've considered**

The logic could instead be moved to `FileFormat::create_physical_plan` implementations, but I think it would be better to keep what is a catalog detail close to the catalog implementation.

FYI @matthewmturner @yjshen @rdettai

Contributor guide

Open the contributing guide

Research direction

Start by reading ListingTable and TableProvider::scan, then review the FileFormat::create_physical_plan implementations and the dependency in #2289. Trace how Hive partition values are currently projected and how ProjectionExec and UnionExec are assembled. Done means partition projection is handled by ListingTable rather than file-format-specific physical operators, with the proposed plan shape working for multiple partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.