[Python][Parquet] Improve usability of ParquetDataset
- 主要言語
- C++
- スター
- 17.1k
- フォーク
- 4.3k
- 平均マージ
- 3日 23時間
- マージ済み PR(30日)
- 101
説明
In e.g. `pd.read_parquet` implementation, we currently wrap `pq.read_table`. However, I would like to use the dataset implementation to get access to some more features (e.g. reading only the first few rows, defining a filter _after_ opening (discovering the schema) the dataset).
That runs into some issues:
- `pq.read_table()` uses ParquetDataset under the hood, but still has a fallback to plain `ParquetFile().read()` when `pyarrow.dataset` module is not available. While this will definitely be uncommon, I have no clue how important it is to keep supporting this (also on the pandas side, since this is currently a "feature" of `pandas.read_parquet` by means of using `pq.read_table`).
- I would prefer using `pq.ParquetDataset` over `pyarrow.dataset`, because the parquet version is compatible with `pq.read_table` and does all the translation to the `pyarrow.dataset` API for us (constructing the file format object, passing the various keywords in the correct place, mapping some naming differences, etc)
- But `ParquetDataset` is also limited, and currently I am essentially using it as a constructor to then access the underlying `._dataset` (the `pyarrow.dataset` dataset object)
Some ideas that I was having:
- Add a new `pq.open_dataset()`-like function that also has the boilerplate to construct the `pyarrow.dataset` object, but returns that instead of the `ParquetDataset` wrapper
- Would it be technically possible to let `pq.ParquetDataset` inherit from `pyarrow.dataset.Dataset`, so you get those methods that way, while keeping it back-compat
- Add new methods to `pq.ParquetDataset` (`to_table()`, `head()`, etc) to make it look more like a `pyarrow.dataset.Dataset`, but without actually inheriting from it
- Simply "officially" expose the underlying dataset, so that I don't have to use the private `_dataset` attribute
コントリビューションガイド
調査の方向性
まず pandas.read_parquet と pq.read_table の統合を確認し、次に pq.ParquetDataset とその基盤となる _dataset を pyarrow.dataset API と併せて調査します。提案されている open_dataset 関数、継承、追加されたメソッド、および公開データセットの公開方法を比較します。完了条件は、適切な API カバレッジを備えた互換性のある使いやすさの方向性を 1 つ選択し、文書化することです。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- data-engineering
- issue の種類
- 機能追加
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 活発
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100