Add Spark-compatible Parquet reader
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I would like to add a Spark-compatible Parquet reader to the `datafusion-spark` crate, based on functionally that already exists in Comet.
The Parquet reader should support per-Spark-version behavior. For example, there were significant differences between Spark 3.x and 4.x with regards to type widening and handling of `TimestampNTZ`.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the datafusion-spark crate and locating the existing Comet functionality referenced in the issue. Compare the Spark 3.x and 4.x behavior for type widening and TimestampNTZ, then define done as a Parquet reader that applies the correct behavior for each supported Spark version.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100