apache / apache/gluten

Support input file expressions for all table formats

Open
#10,840 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

In the initial implementation of the input file expression support, https://github.com/apache/incubator-gluten/pull/6021, it covered only Hive tables. After that, the Iceberg table support was added, https://github.com/apache/incubator-gluten/pull/10831. Similar implementation is needed for Delta as it extends FileSourceScanExecTransformerBase and the pushdown rule only covers FileSourceScanExecTransformer like below,
https://github.com/apache/incubator-gluten/blob/fd155d23b899843d019853fa55a15a0606f0f6ff/gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PushDownInputFileExpression.scala#L117

Indeed, all other table formats need to be checked and implemented if missing.

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PushDownInputFileExpression.scala, especially the rule around line 117, and review the linked Hive and Iceberg implementations. Check how Delta and the remaining table formats extend the relevant scan classes; done means input file expressions are supported wherever they are currently missing.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.