apache / apache/gluten

[CH] support customized file scan which does not support transform

Open
#1,218 0 comments 0 reactions 0 assignees View on GitHub
enhancement stale
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When we use customized file scan( not support transform ) , the output spark columnar format data cannot be passed to a transform supported exec directly.

**Describe the solution you'd like**
A columnar(spark) to columnar(backend native) plan ts needed between file scan and its transform supported parent

**Describe alternatives you've considered**
Otherwise we can use a step combined of C2R and R2C instead
For C2R and R2C are already implemented

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing C2R and R2C implementations mentioned in the issue, then trace how a customized file scan connects to a transform-supported parent. The requested result is a columnar Spark-to-native plan between those operators without requiring a transform-unsupported scan.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala, spark
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.