apache / apache/gluten

[VL] Pushdown scan and filter to remote Storage

Open
#8,856 0 comments 0 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

One common issue on cloud is the remote storage throughput. Currently we use rowgroup skipping to reduce the data size.
The plan is to pushdown the scan and filter as substrait plan to remote storage, get the data in arrow format and input into pipeline.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are identified. Start by locating the existing rowgroup-skipping path, then determine how scan and filter operations could be represented as a Substrait plan for remote storage and how Arrow-format data would re-enter the pipeline; done requires an agreed design and working end-to-end pushdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.