Can Parquet and Orc sources support path expressions on the schema projection?
- Lingua principale
- Scala
- Stelle
- 147
- Fork
- 32
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
- I can successfully write out rows with structs like so with Parquet:
```scala
val personDetailsStruct = Field.createStructField("PERSON_DETAILS",
Seq(
Field("NAME", StringType),
Field("AGE", IntType.Signed),
Field("SALARY", DecimalType(Precision(38), Scale(5))),
Field("CREATION_TIME", TimestampMillisType)
)
)
val schema = StructType(personDetailsStruct)
val rows = Vector(
Vector(Vector("Fred", 50, BigDecimal("50000.99000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Gary", 50, BigDecimal("20000.34000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Alice", 50, BigDecimal("99999.98000"), new Timestamp(System.currentTimeMillis())))
)
Frame.fromValues(schema, rows)
.to(ParquetSink(parquetFilePath))
```
- But **fails** to read it back using the path expression **PERSON_DETAILS.NAME**:
```scala
ParquetSource(parquetFilePath)
.withProjection("PERSON_DETAILS.NAME")
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note path expressions **work** with **withPredicate**
```scala
ParquetSource(parquetFilePath)
.withPredicate(Predicate.or(Predicate.equals("PERSON_DETAILS.NAME", "Alice"), Predicate.equals("PERSON_DETAILS.NAME", "Gary")))
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note a **Hive** table pointing at this file works using the following **HiveQL** syntax:
```sql
hive> select person_details.name, person_details.age
> from struct_person
> where person_details.name in ('Alice', 'Gary' );
OK
Gary 50
Alice 50
Time taken: 0.067 seconds, Fetched: 2 row(s)
hive>
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.