Can Parquet and Orc sources support path expressions on the schema projection?
- Lenguaje dominante
- Scala
- Estrellas
- 147
- Forks
- 32
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
- I can successfully write out rows with structs like so with Parquet:
```scala
val personDetailsStruct = Field.createStructField("PERSON_DETAILS",
Seq(
Field("NAME", StringType),
Field("AGE", IntType.Signed),
Field("SALARY", DecimalType(Precision(38), Scale(5))),
Field("CREATION_TIME", TimestampMillisType)
)
)
val schema = StructType(personDetailsStruct)
val rows = Vector(
Vector(Vector("Fred", 50, BigDecimal("50000.99000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Gary", 50, BigDecimal("20000.34000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Alice", 50, BigDecimal("99999.98000"), new Timestamp(System.currentTimeMillis())))
)
Frame.fromValues(schema, rows)
.to(ParquetSink(parquetFilePath))
```
- But **fails** to read it back using the path expression **PERSON_DETAILS.NAME**:
```scala
ParquetSource(parquetFilePath)
.withProjection("PERSON_DETAILS.NAME")
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note path expressions **work** with **withPredicate**
```scala
ParquetSource(parquetFilePath)
.withPredicate(Predicate.or(Predicate.equals("PERSON_DETAILS.NAME", "Alice"), Predicate.equals("PERSON_DETAILS.NAME", "Gary")))
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note a **Hive** table pointing at this file works using the following **HiveQL** syntax:
```sql
hive> select person_details.name, person_details.age
> from struct_person
> where person_details.name in ('Alice', 'Gary' );
OK
Gary 50
Alice 50
Time taken: 0.067 seconds, Fetched: 2 row(s)
hive>
```
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Look at the ParquetSource and OrcSource classes to see how projection is handled. The issue shows that path expressions work for predicates but not for schema projection. Start by examining the projection parsing logic and compare it with predicate parsing. Check if the projection logic correctly handles nested struct fields like PERSON_DETAILS.NAME. Run the provided test code to reproduce the failure.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- scala
- Área
- data-engineering
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Estancado
- Claridad
- Bien especificado
- Aptitud para principiantes
- 45/100