apache / apache/datafusion-ballista
Ballista should serialize Parquet statistics
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When the Ballista scheduler or executor deserializes a ParquetExec it collects the statistics again and this is redundant. We should serialize the statistics to avoid this extra work.
**Describe the solution you'd like**
Add Parquet statistics to serde module.
**Describe alternatives you've considered**
N/A
**Additional context**
N/A
Contributor guide
Research direction
Start in the serde module and trace how ParquetExec is serialized and deserialized by the Ballista scheduler or executor. Done means Parquet statistics are included in serialization and deserialization no longer redundantly collects them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100