51zero / 51zero/eel-sdk

Can Parquet and Orc sources support path expressions on the schema projection?

Open
#232 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
147
Forks
32
PR merge metrics
No merged PRs in 30d

Description

- I can successfully write out rows with structs like so with Parquet:
```scala
val personDetailsStruct = Field.createStructField("PERSON_DETAILS",
Seq(
Field("NAME", StringType),
Field("AGE", IntType.Signed),
Field("SALARY", DecimalType(Precision(38), Scale(5))),
Field("CREATION_TIME", TimestampMillisType)
)
)
val schema = StructType(personDetailsStruct)

val rows = Vector(
Vector(Vector("Fred", 50, BigDecimal("50000.99000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Gary", 50, BigDecimal("20000.34000"), new Timestamp(System.currentTimeMillis()))),
Vector(Vector("Alice", 50, BigDecimal("99999.98000"), new Timestamp(System.currentTimeMillis())))
)
Frame.fromValues(schema, rows)
.to(ParquetSink(parquetFilePath))
```
- But **fails** to read it back using the path expression **PERSON_DETAILS.NAME**:
```scala
ParquetSource(parquetFilePath)
.withProjection("PERSON_DETAILS.NAME")
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note path expressions **work** with **withPredicate**

```scala
ParquetSource(parquetFilePath)
.withPredicate(Predicate.or(Predicate.equals("PERSON_DETAILS.NAME", "Alice"), Predicate.equals("PERSON_DETAILS.NAME", "Gary")))
.toFrame()
.collect()
.foreach(row => println(row))
```
- Note a **Hive** table pointing at this file works using the following **HiveQL** syntax:
```sql
hive> select person_details.name, person_details.age
> from struct_person
> where person_details.name in ('Alice', 'Gary' );
OK
Gary 50
Alice 50
Time taken: 0.067 seconds, Fetched: 2 row(s)
hive>
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Look at the ParquetSource and OrcSource classes to see how projection is handled. The issue shows that path expressions work for predicates but not for schema projection. Start by examining the projection parsing logic and compare it with predicate parsing. Check if the projection logic correctly handles nested struct fields like PERSON_DETAILS.NAME. Run the provided test code to reproduce the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.