Macro based Scala APIs for Predicate and Projection push down
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
There has been some experiments with Scala macros to make Predicate and Projection APIs in Scala.
In particular by `[~neville]`:
https://github.com/nevillelyh/parquet-avro-extra/blob/master/parquet-avro-examples/src/main/scala/me/lyh/parquet/avro/examples/ExampleApp.scala
https://github.com/nevillelyh/parquet-avro-extra/commit/090f8d1d1d7d44489bb71e04465d4a12a46432c9#diff-89dca6adcdf105c6063661de43931c29R15
There have also been some discussion at Twitter.
**Reporter**: [Julien Le Dem](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=julienledem) / @julienledem
**Note**: *This issue was originally created as [PARQUET-218](https://issues.apache.org/jira/browse/PARQUET-218). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked parquet-avro-extra ExampleApp.scala and the referenced commit, then compare their Scala macro experiments with the current parquet-java APIs. The issue is complete when a defined macro-based approach for Predicate and Projection push down is agreed and implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100