apache / apache/parquet-java

Decouple parquet-hadoop module from hadoop-mapreduce-client-core

オープン
#3,780 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
Type: enhancement
主要言語
Java
スター
3.1k
フォーク
1.6k
平均マージ
3日 12時間
マージ済み PR(30日)
33

説明

### Describe the enhancement requested

Decouple `ParquetReadOptions` from the legacy Hadoop `ParquetInputFormat`/`FileInputFormat` classes, which currently force pulling in the [`hadoop-mapreduce-client-core`](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core) dependency (and its transitive JARs).

## Summary

To instantiate a `org.apache.parquet.hadoop.ParquetReader` we need to use `org.apache.parquet.ParquetReadOptions`, which references a set of keys located in `org.apache.parquet.hadoop.ParquetInputFormat` and calls a static `getFilter` method declared also on `ParquetInputFormat`, which `extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat`.

`ParquetInputFormat` is part of the `parquet-hadoop` module, while `FileInputFormat` is declared in the `hadoop-mapreduce-client-core` JAR from the Hadoop project.

Because `ParquetReader` (via `ParquetReadOptions`) needs to call `ParquetInputFormat.getFilter(...)`, the JVM is forced to initialize `ParquetInputFormat`, and initializing a class triggers the loading and initialization of its superclass (`FileInputFormat`) along with its entire transitive dependency graph (`org.apache.hadoop.mapreduce.*`). For code that only needs to read a plain Parquet file (and not the MapReduce input-format machinery), this pulls an unwanted, heavy Hadoop-`mapreduce` dependency into the classpath and link set. `AvroParquetReader` and `ProtoParquetReader` extend from `ParquetReader` and have the same issue.

`ParquetInputFormat` has three responsibilities:
* define a set of property keys as constants
* deserialize the filter predicates from a configuration value
* support the integration of Parquet files into Hadoop MapReduce

This issue proposes to extract the first two responsibilities into two new Hadoop-agnostic classes, in the `org.apache.parquet.conf` package:
* `ParquetInputProperties`: the property-key constants only
* `ParquetInputFilters`: the filter deserialization logic
so that the configuration can be used without ever loading `FileInputFormat` and its transitive dependencies.

```
ParquetReader ← org.apache.parquet.hadoop (parquet-hadoop)
│ uses

ParquetReadOptions ← org.apache.parquet (parquet-hadoop)
│ (static import keys + getFilter call)

ParquetInputFormat.getFilter(...) ← org.apache.parquet.hadoop (parquet-hadoop)
│ extends

FileInputFormat ← org.apache.hadoop.mapreduce.lib.input (hadoop-mapreduce-client-core)
│ extends

InputFormat ← org.apache.hadoop.mapreduce.lib.input (hadoop-mapreduce-client-core)
│ (transitive)

{ InputSplit, JobContext, TaskAttemptContext,
RecordReader, ... } ← org.apache.hadoop.mapreduce* (hadoop-mapreduce-client-core)
```

With the new `ParquetInputProperties` / `ParquetInputFilters`, building `ParquetReadOptions` no longer touches any `org.apache.hadoop.mapreduce` type, so consumers not related to Hadoop avoid transitively including the MapReduce dependency.

## Public API / Behavioral change

No behavior change. This is a refactoring:

- New classes `org.apache.parquet.conf.ParquetInputProperties` and
`org.apache.parquet.conf.ParquetInputFilters` (in `parquet-hadoop`) hold the constants and the
`ParquetConfiguration`-based filter resolution respectively.
- The legacy `org.apache.hadoop.conf.Configuration`-based entry points remain on
`ParquetInputFormat` (they are used only via the legacy MapReduce path) and are kept for binary /
source compatibility.
- All pre-existing constants on `ParquetInputFormat` are now `@Deprecated` and delegate to the new
classes; source and binary compatibility are preserved.

## Acceptance criteria

- `ParquetReadOptions` (used for plain file reads) references only `ParquetInputProperties` / `ParquetInputFilters`, never `ParquetInputFormat` / `org.apache.hadoop.mapreduce.InputFormat`.
- Loading `ParquetInputProperties` / `ParquetInputFilters` / `ParquetReadOptions` does not initialize `FileInputFormat`.
- All existing tests still pass (`./mvnw test`).

### Component(s)

Core

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

parquet-hadoop モジュールの ParquetReadOptions と ParquetInputFormat から始め、既存の定数とフィルター解決ロジックを追跡します。要求された org.apache.parquet.conf クラスを導入しつつ、既存の ParquetInputFormat エントリーポイントと互換性を維持します。新しいクラスと ParquetReadOptions が FileInputFormat を初期化しないことを確認してから、./mvnw test を実行します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
hadoop, java
領域
data-engineering
issue の種類
リファクタリング
難易度
4/5
見積もり時間
3〜5日
活発さ
活発
明瞭さ
明確に書かれている
初心者へのやさしさ
68/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。