Can Parquet and Orc sources support path expressions on the schema projection?
- 主要言語
- Scala
- スター
- 147
- フォーク
- 32
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
- I can successfully write out rows with structs like so with Parquet:
```scala
val personDetailsStruct = Field.createStructField("PERSON_DETAILS",
Seq(
Field("NAME", StringType),
Field("AGE", IntType.Signed),
Field("SALARY", DecimalType(Precision(38), Scale(5))),
Field("CREATION_TIME", TimestampMillisType)
)
)
val schema = StructType(personDetailsStruct)
val rows = Vector(
Vector(Vector("Fred", 50, BigDecimal("50000.99000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Gary", 50, BigDecimal("20000.34000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Alice", 50, BigDecimal("99999.98000"), new Timestamp(System.currentTimeMillis())))
)
Frame.fromValues(schema, rows)
.to(ParquetSink(parquetFilePath))
```
- But **fails** to read it back using the path expression **PERSON_DETAILS.NAME**:
```scala
ParquetSource(parquetFilePath)
.withProjection("PERSON_DETAILS.NAME")
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note path expressions **work** with **withPredicate**
```scala
ParquetSource(parquetFilePath)
.withPredicate(Predicate.or(Predicate.equals("PERSON_DETAILS.NAME", "Alice"), Predicate.equals("PERSON_DETAILS.NAME", "Gary")))
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note a **Hive** table pointing at this file works using the following **HiveQL** syntax:
```sql
hive> select person_details.name, person_details.age
> from struct_person
> where person_details.name in ('Alice', 'Gary' );
OK
Gary 50
Alice 50
Time taken: 0.067 seconds, Fetched: 2 row(s)
hive>
```
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Look at the ParquetSource and OrcSource classes to see how projection is handled. The issue shows that path expressions work for predicates but not for schema projection. Start by examining the projection parsing logic and compare it with predicate parsing. Check if the projection logic correctly handles nested struct fields like PERSON_DETAILS.NAME. Run the provided test code to reproduce the failure.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- scala
- 領域
- data-engineering
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 停滞
- 明瞭さ
- 明確に書かれている
- 初心者へのやさしさ
- 45/100