apache / apache/arrow

[Python] Extract partition list from pyarrow.dataset.ParquetFileFragment object

Open
#34,212 4 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

A` pyarrow.dataset.ParquetFileFragment` object has two elements: `path` and `partition`. The path can be extracted using `path` attribute but there is no attribute to get the partition.
For example for the following object:
``

`object.path `returns `pq-test/Location=US/Industry=HT/dce9900c46f94ec3a8dca094cf62bd34-0.parquet` but `object.partition` has not been defined.

Also, it seems the order of partition list is reverse.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reading the Python API for `pyarrow.dataset.ParquetFileFragment` and how its `path` and partition information are represented. The issue asks for a way to access the partition and notes that the displayed partition order may be reversed; done means the partition is accessible and its ordering is clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.