apache / apache/gluten

[VL] IcebergScanTransformer.getRootPathsInternal always returns Seq.empty, silently skipping native filesystem scheme validation

Open Beginner friendly
#12,712 0 comments 1 reaction 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

[Expected behavior]: When `spark.gluten.sql.columnar.batchscan=true` and reading from an Iceberg table, Gluten's native scan validation (`VeloxBackend.validateScanExec` → `validateScheme`) should check whether the table's storage filesystem scheme is supported by the registered native (Velox) filesystems, and fall back to vanilla Spark if not — the same way it does for DSv1/DSv2 file-based scans via `FileIndex.rootPaths`.

[Actual behavior]: `IcebergScanTransformer.getRootPathsInternal` (`gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala:179`) is hardcoded to return `Seq.empty`, behind a `// TODO: get root paths from table.` comment:

```scala
// TODO: get root paths from table.
override def getRootPathsInternal: Seq[String] = Seq.empty
```

`validateScheme()` in `VeloxBackend.scala` treats an empty `rootPaths` as "nothing to check" (only validates `if (filteredRootPaths.nonEmpty && ...)`), so scheme validation is silently skipped for every Iceberg scan regardless of the table's actual filesystem. Gluten will not correctly fall back to vanilla Spark for Iceberg tables on a scheme unsupported by the native build; any resulting failure instead surfaces later inside native code, in a much less clear form.

Suggested fix — return the Iceberg table's base location, mirroring how DSv1/DSv2 scans expose their root paths:

```scala
override def getRootPathsInternal: Seq[String] = {
table match {
case t: SparkTable => Seq(t.table().location())
case _ => Seq.empty
}
}
```

We found this while investigating an unrelated S3A native-scan issue. We have a small patch ready (verified via local compile) and are happy to open a PR referencing this issue, along with a regression test.

Possibly related (unconfirmed): #7528, a native SIGSEGV on GCS + Iceberg with no established root cause.

This issue was written with the assistance of AI (used to help organize and phrase the investigation notes).

### Gluten version

main branch

### Spark version

Spark-4.0.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start in gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala at getRootPathsInternal, then trace VeloxBackend.validateScanExec and validateScheme. Use the existing DSv1/DSv2 root-path behavior as a reference and add or run a regression test showing that an unsupported Iceberg filesystem scheme falls back to vanilla Spark while a supported scheme remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.