[VL] Iceberg REST-catalog vended credentials (Polaris etc.) never reach the native S3 reader — native scans 403 while vanilla Spark reads fine
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
When a Spark job reads an Apache Iceberg table through a REST catalog that uses **credential vending** (`X-Iceberg-Access-Delegation: vended-credentials`, e.g. Apache Polaris), Gluten's native scan path cannot read the table's data files: every native `TableScan` fails with S3 403 while the same query on vanilla Spark succeeds.
```
VeloxRuntimeError ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied'
```
The failure is deterministic (task retries exhaust → job abort) whenever the compute's own IAM identity has no direct grant on the warehouse bucket — which is the *point* of credential vending: the catalog returns per-table, short-TTL, prefix-scoped STS credentials on `loadTable`, and only the JVM `S3FileIO` ever sees them.
**Mechanism**
- JVM path (works): Iceberg REST `loadTable` puts `s3.access-key-id` / `s3.secret-access-key` / `s3.session-token` on the table's FileIO properties; `S3FileIO` reads data files with them.
- Native path (fails): `GlutenIcebergSourceUtil.genSplitInfo` extracts only file paths/starts/lengths/deletes into the `LocalFiles` split; Velox's S3 filesystem builds its client from the global `hive.s3.*` config + the AWS default credential chain. The vended credentials never leave the JVM. There is no per-table (or even session-token) credential channel: velox `S3Config` has no session-token key at all, so an STS triple is not even expressible statically.
Reproduced on Gluten v1.6.0 (Spark 3.5.2, iceberg-spark-runtime 1.6.0, Apache Polaris with STS vending). Code on `main` has the same gap: the iceberg module still reads no FileIO properties, and the `fs.s3a.*` → `hive.s3.*` static mapping has no session-token entry.
Related but distinct: #10113 / #8170 (session-level / custom credential providers not reaching native) are session-global; vended credentials are **per-table**, so two tables in one bucket can carry different credentials in one query — a bucket- or session-scoped channel cannot express this.
**Prior art:** apache/datafusion-comet hit exactly this and fixed it in two stages — static extraction of the vended credentials from the table's FileIO properties into the native scan payload (comet#3523), then a pluggable per-path credential provider for refresh (comet#4309).
### How we fixed it (patch set available, happy to upstream)
We run Gluten v1.6.0 with three bundle patches that make native scans of vended-credential tables work end-to-end, validated against Apache Polaris:
1. **JVM extract + carry:** at split planning, read the scan table's `FileIO.properties()`; when a vended `s3.access-key-id`/`s3.secret-access-key` pair is present, attach it (+ session token/expiry/endpoint/region) with the table location to a new table-scoped `ReadRel.LocalFiles.read_properties` map (one `LocalFiles` is single-table by construction). Config-gated with a kill switch whose *off* branch fails native validation → clean vanilla fallback instead of 403s.
2. **Native install:** parse the map onto gluten's `SplitInfo`; union all scans' entries into a `filesystems::TokenProvider` implementation (longest-prefix match of file path against normalized table locations → per-table resolution, multi-table-same-bucket safe) passed to `QueryCtx::create`. The velox line Gluten pins (IBM/velox `dft-*` branches) already threads `fsTokenProvider()` through `FileHandleKey` (credential identity keys the handle cache) into `openFileForRead` for both data and positional-delete files.
3. **Velox S3 consumption:** `S3FileSystem::openFileForRead/Write` consult `FileOptions::tokenProvider` (new `S3AccessTokenKey`/`S3AccessToken` types) and serve the open from a per-credential-fingerprint client cache; `S3Config` gains `hive.s3.session-token` (per-bucket capable) so static STS credentials are expressible too.
Known limitation (same as comet#3523's explicit non-goal): credentials are snapshotted at planning; refresh-on-open is a follow-up.
If maintainers are interested I can open PRs for the gluten side (and the velox piece to the pinned velox line). Adoption of REST-catalog credential vending is growing quickly (Polaris, Unity, Glue IRC all vend), so I'd expect this class to hit more users as `use_gluten`-style rollouts meet lakehouse catalogs.
### Spark version
Spark-3.5.x
### Spark configurations
spark.plugins=org.apache.gluten.GlutenPlugin
spark.sql.catalog.=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog..type=rest
spark.sql.catalog..header.X-Iceberg-Access-Delegation=vended-credentials
### System information
Gluten v1.6.0 (velox backend, IBM/velox dft-2026_02_06), Spark 3.5.2, JDK 17, iceberg-spark-runtime-3.5_2.12:1.6.0, aws-sdk-cpp 1.11.655, arm64 + x86_64
### Relevant logs
```
VeloxRuntimeError: ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied'
(HTTP 403 GetObject on the REST-catalog warehouse bucket; IAM: the pod role, not the vended credentials)
```
Contributor guide
Research direction
Start at GlutenIcebergSourceUtil.genSplitInfo and trace how LocalFiles reaches SplitInfo and QueryCtx::create. Then inspect S3FileSystem::openFileForRead/Write and S3Config on the pinned Velox line. Done means native scans can use table-scoped vended credentials, including delete-file reads and multi-table queries, while unsupported cases fall back cleanly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, java
- Domain
- backend, cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100