apache / apache/gluten

[VL] Iceberg REST-catalog vended credentials (Polaris etc.) never reach the native S3 reader — native scans 403 while vanilla Spark reads fine

Open
#12,701 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

When a Spark job reads an Apache Iceberg table through a REST catalog that uses **credential vending** (`X-Iceberg-Access-Delegation: vended-credentials`, e.g. Apache Polaris), Gluten's native scan path cannot read the table's data files: every native `TableScan` fails with S3 403 while the same query on vanilla Spark succeeds.

```
VeloxRuntimeError ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied'
```

The failure is deterministic (task retries exhaust → job abort) whenever the compute's own IAM identity has no direct grant on the warehouse bucket — which is the *point* of credential vending: the catalog returns per-table, short-TTL, prefix-scoped STS credentials on `loadTable`, and only the JVM `S3FileIO` ever sees them.

**Mechanism**

- JVM path (works): Iceberg REST `loadTable` puts `s3.access-key-id` / `s3.secret-access-key` / `s3.session-token` on the table's FileIO properties; `S3FileIO` reads data files with them.
- Native path (fails): `GlutenIcebergSourceUtil.genSplitInfo` extracts only file paths/starts/lengths/deletes into the `LocalFiles` split; Velox's S3 filesystem builds its client from the global `hive.s3.*` config + the AWS default credential chain. The vended credentials never leave the JVM. There is no per-table (or even session-token) credential channel: velox `S3Config` has no session-token key at all, so an STS triple is not even expressible statically.

Reproduced on Gluten v1.6.0 (Spark 3.5.2, iceberg-spark-runtime 1.6.0, Apache Polaris with STS vending). Code on `main` has the same gap: the iceberg module still reads no FileIO properties, and the `fs.s3a.*` → `hive.s3.*` static mapping has no session-token entry.

Related but distinct: #10113 / #8170 (session-level / custom credential providers not reaching native) are session-global; vended credentials are **per-table**, so two tables in one bucket can carry different credentials in one query — a bucket- or session-scoped channel cannot express this.

**Prior art:** apache/datafusion-comet hit exactly this and fixed it in two stages — static extraction of the vended credentials from the table's FileIO properties into the native scan payload (comet#3523), then a pluggable per-path credential provider for refresh (comet#4309).

### How we fixed it (patch set available, happy to upstream)

We run Gluten v1.6.0 with three bundle patches that make native scans of vended-credential tables work end-to-end, validated against Apache Polaris:

1. **JVM extract + carry:** at split planning, read the scan table's `FileIO.properties()`; when a vended `s3.access-key-id`/`s3.secret-access-key` pair is present, attach it (+ session token/expiry/endpoint/region) with the table location to a new table-scoped `ReadRel.LocalFiles.read_properties` map (one `LocalFiles` is single-table by construction). Config-gated with a kill switch whose *off* branch fails native validation → clean vanilla fallback instead of 403s.
2. **Native install:** parse the map onto gluten's `SplitInfo`; union all scans' entries into a `filesystems::TokenProvider` implementation (longest-prefix match of file path against normalized table locations → per-table resolution, multi-table-same-bucket safe) passed to `QueryCtx::create`. The velox line Gluten pins (IBM/velox `dft-*` branches) already threads `fsTokenProvider()` through `FileHandleKey` (credential identity keys the handle cache) into `openFileForRead` for both data and positional-delete files.
3. **Velox S3 consumption:** `S3FileSystem::openFileForRead/Write` consult `FileOptions::tokenProvider` (new `S3AccessTokenKey`/`S3AccessToken` types) and serve the open from a per-credential-fingerprint client cache; `S3Config` gains `hive.s3.session-token` (per-bucket capable) so static STS credentials are expressible too.

Known limitation (same as comet#3523's explicit non-goal): credentials are snapshotted at planning; refresh-on-open is a follow-up.

If maintainers are interested I can open PRs for the gluten side (and the velox piece to the pinned velox line). Adoption of REST-catalog credential vending is growing quickly (Polaris, Unity, Glue IRC all vend), so I'd expect this class to hit more users as `use_gluten`-style rollouts meet lakehouse catalogs.

### Spark version

Spark-3.5.x

### Spark configurations

spark.plugins=org.apache.gluten.GlutenPlugin
spark.sql.catalog.=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog..type=rest
spark.sql.catalog..header.X-Iceberg-Access-Delegation=vended-credentials

### System information

Gluten v1.6.0 (velox backend, IBM/velox dft-2026_02_06), Spark 3.5.2, JDK 17, iceberg-spark-runtime-3.5_2.12:1.6.0, aws-sdk-cpp 1.11.655, arm64 + x86_64

### Relevant logs

```
VeloxRuntimeError: ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied'
(HTTP 403 GetObject on the REST-catalog warehouse bucket; IAM: the pod role, not the vended credentials)
```

Contributor guide

Open the contributing guide

Research direction

Start at GlutenIcebergSourceUtil.genSplitInfo and trace how LocalFiles reaches SplitInfo and QueryCtx::create. Then inspect S3FileSystem::openFileForRead/Write and S3Config on the pinned Velox line. Done means native scans can use table-scoped vended credentials, including delete-file reads and multi-table queries, while unsupported cases fall back cleanly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, java
Domain
backend, cloud, data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.