Can Parquet and Orc sources support path expressions on the schema projection?
- Lingua principale
- Scala
- Stelle
- 147
- Fork
- 32
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
- I can successfully write out rows with structs like so with Parquet:
```scala
val personDetailsStruct = Field.createStructField("PERSON_DETAILS",
Seq(
Field("NAME", StringType),
Field("AGE", IntType.Signed),
Field("SALARY", DecimalType(Precision(38), Scale(5))),
Field("CREATION_TIME", TimestampMillisType)
)
)
val schema = StructType(personDetailsStruct)
val rows = Vector(
Vector(Vector("Fred", 50, BigDecimal("50000.99000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Gary", 50, BigDecimal("20000.34000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Alice", 50, BigDecimal("99999.98000"), new Timestamp(System.currentTimeMillis())))
)
Frame.fromValues(schema, rows)
.to(ParquetSink(parquetFilePath))
```
- But **fails** to read it back using the path expression **PERSON_DETAILS.NAME**:
```scala
ParquetSource(parquetFilePath)
.withProjection("PERSON_DETAILS.NAME")
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note path expressions **work** with **withPredicate**
```scala
ParquetSource(parquetFilePath)
.withPredicate(Predicate.or(Predicate.equals("PERSON_DETAILS.NAME", "Alice"), Predicate.equals("PERSON_DETAILS.NAME", "Gary")))
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note a **Hive** table pointing at this file works using the following **HiveQL** syntax:
```sql
hive> select person_details.name, person_details.age
> from struct_person
> where person_details.name in ('Alice', 'Gary' );
OK
Gary 50
Alice 50
Time taken: 0.067 seconds, Fetched: 2 row(s)
hive>
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Look at the ParquetSource and OrcSource classes to see how projection is handled. The issue shows that path expressions work for predicates but not for schema projection. Start by examining the projection parsing logic and compare it with predicate parsing. Check if the projection logic correctly handles nested struct fields like PERSON_DETAILS.NAME. Run the provided test code to reproduce the failure.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- scala
- Ambito
- data-engineering
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 45/100