[VL] IcebergScanTransformer.getRootPathsInternal always returns Seq.empty, silently skipping native filesystem scheme validation
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
[Expected behavior]: When `spark.gluten.sql.columnar.batchscan=true` and reading from an Iceberg table, Gluten's native scan validation (`VeloxBackend.validateScanExec` → `validateScheme`) should check whether the table's storage filesystem scheme is supported by the registered native (Velox) filesystems, and fall back to vanilla Spark if not — the same way it does for DSv1/DSv2 file-based scans via `FileIndex.rootPaths`.
[Actual behavior]: `IcebergScanTransformer.getRootPathsInternal` (`gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala:179`) is hardcoded to return `Seq.empty`, behind a `// TODO: get root paths from table.` comment:
```scala
// TODO: get root paths from table.
override def getRootPathsInternal: Seq[String] = Seq.empty
```
`validateScheme()` in `VeloxBackend.scala` treats an empty `rootPaths` as "nothing to check" (only validates `if (filteredRootPaths.nonEmpty && ...)`), so scheme validation is silently skipped for every Iceberg scan regardless of the table's actual filesystem. Gluten will not correctly fall back to vanilla Spark for Iceberg tables on a scheme unsupported by the native build; any resulting failure instead surfaces later inside native code, in a much less clear form.
Suggested fix — return the Iceberg table's base location, mirroring how DSv1/DSv2 scans expose their root paths:
```scala
override def getRootPathsInternal: Seq[String] = {
table match {
case t: SparkTable => Seq(t.table().location())
case _ => Seq.empty
}
}
```
We found this while investigating an unrelated S3A native-scan issue. We have a small patch ready (verified via local compile) and are happy to open a PR referencing this issue, along with a regression test.
Possibly related (unconfirmed): #7528, a native SIGSEGV on GCS + Iceberg with no established root cause.
This issue was written with the assistance of AI (used to help organize and phrase the investigation notes).
### Gluten version
main branch
### Spark version
Spark-4.0.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start in gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala at getRootPathsInternal, then trace VeloxBackend.validateScanExec and validateScheme. Use the existing DSv1/DSv2 root-path behavior as a reference and add or run a regression test showing that an unsupported Iceberg filesystem scheme falls back to vanilla Spark while a supported scheme remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100