[VL] Pushdown scan and filter to remote Storage
Open
enhancement
parquet
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
One common issue on cloud is the remote storage throughput. Currently we use rowgroup skipping to reduce the data size.
The plan is to pushdown the scan and filter as substrait plan to remote storage, get the data in arrow format and input into pipeline.
Contributor guide
Research direction
No files, tests, or entry points are identified. Start by locating the existing rowgroup-skipping path, then determine how scan and filter operations could be represented as a Substrait plan for remote storage and how Arrow-format data would re-enter the pipeline; done requires an agreed design and working end-to-end pushdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100