Spark: remove_orphan_files lists the table location with session credentials, not the catalog's

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

We run a single Spark Connect cluster (Spark 4.1.2) serving several tenants. Each tenant has its own Glue catalog in its own AWS account, reached with `AssumeRoleAwsClientFactory`:

```
spark.sql.catalog.tenant org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.tenant.type glue
spark.sql.catalog.tenant.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.tenant.client.factory org.apache.iceberg.aws.AssumeRoleAwsClientFactory
spark.sql.catalog.tenant.client.assume-role.arn arn:aws:iam::111122223333:role/lakehouse-access
spark.sql.catalog.tenant.client.assume-role.region us-east-2
spark.sql.catalog.tenant.glue.id 111122223333
```

The cluster itself runs on EKS as an IRSA role in a different account, which deliberately has no access to the tenant buckets:

```
spark.hadoop.fs.s3.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.aws.credentials.provider com.amazonaws.auth.WebIdentityTokenCredentialsProvider
```

Nightly maintenance calls `expire_snapshots` and then `remove_orphan_files` on each table. The first one works, the second one gets a 403, seconds later, on the same table:

```sql
-- ok
CALL tenant.system.expire_snapshots(table => 'db.my_table', older_than => TIMESTAMP '2026-08-21 00:00:00');

-- 403
CALL tenant.system.remove_orphan_files(table => 'db.my_table', older_than => TIMESTAMP '2026-08-25 00:00:00');
```

```
java.io.UncheckedIOException: java.nio.file.AccessDeniedException:
s3://my-lakehouse-bucket/iceberg_tables/my_table: listObjects() on s3://my-lakehouse-bucket/iceberg_tables/my_table:
software.amazon.awssdk.services.s3.model.AccessDeniedException:
User: arn:aws:sts::444455556666:assumed-role/spark-cluster-irsa/... is not authorized to perform:
s3:ListBucket on resource: "arn:aws:s3:::my-lakehouse-bucket" (Service: S3, Status Code: 403)

at org.apache.iceberg.util.FileSystemWalker.listDirRecursivelyWithHadoop(FileSystemWalker.java:150)
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.listedFileDS(DeleteOrphanFilesSparkAction.java:440)
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.doExecute(DeleteOrphanFilesSparkAction.java:257)
at org.apache.iceberg.spark.procedures.RemoveOrphanFilesProcedure.call(RemoveOrphanFilesProcedure.java:153)
```

The principal in that error is the cluster's own IRSA role, which appears nowhere in the catalog config. So the listing is not using the catalog's credentials, while the rest of the procedure is. The cause is that the action takes its Hadoop config from the session and drops the catalog it came from:

```java
// DeleteOrphanFilesSparkAction.java:139
this.hadoopConf = new SerializableConfiguration(spark.sessionState().newHadoopConf());
```

`newHadoopConf()` only sees `spark.hadoop.*`. Everything else in the procedure goes through `table.io()`, which the catalog built with the assume-role factory, which is why `expire_snapshots` is fine.

I found #11541, which describes this exact coupling, and PR #12254 which closed it by adding the `FileIO` listing path. But that path is opt-in (`prefix_listing`, `usePrefixListing` defaults to `false` at `DeleteOrphanFilesSparkAction.java:133`), so the Hadoop path still behaves this way and it is what you get unless you know to ask for the other one.

`prefix_listing => true` does fix the credentials for us, but we can't use it: it lists serially on the driver and then does `parallelize(matchingFiles, 1)` (#16932, #17387). On Spark Connect there's an extra ceiling, since that single partition puts the whole listing under `spark.rpc.message.maxSize`, which is a server startup setting we can't raise from a client session. So we're stuck with the Hadoop path, and the Hadoop path has the wrong credentials.

What took me longest to work out is that there is no per-catalog knob for this at all. `SparkUtil.hadoopConfCatalogOverrides(SparkSession, catalogName)` already exists and applies `spark.sql.catalog..hadoop.*` on top of the session config, which is exactly what's needed here, but it only has two callers, both in `SparkCatalog` (`SparkCatalog.java:142` and `:717`). No action or procedure uses it. So setting `spark.sql.catalog.tenant.hadoop.fs.s3a.*` is accepted, looks right, and is then ignored by the walk. I assumed I had a typo for a while.

Proposal: build the action's Hadoop config with `SparkUtil.hadoopConfCatalogOverrides(spark, catalogName)` instead of `spark.sessionState().newHadoopConf()`, and have that method derive the S3A settings from the catalog's own `client.assume-role.arn`, so the role only has to be declared once:

- `fs.s3a.aws.credentials.provider` becomes `AssumedRoleCredentialProvider`
- `fs.s3a.assumed.role.arn` becomes the catalog's arn
- whatever identity the session already resolved stays on as `fs.s3a.assumed.role.credentials.provider`, so it is still what signs the AssumeRole call. Without that step S3A falls back to its `SimpleAWSCredentialsProvider` default and IRSA or instance-profile clusters end up with no credentials for the STS call at all.

Because that config is per catalog there's nothing to scope by bucket, unlike the session-wide workaround below. `spark.sql.catalog..hadoop.*` is copied over the derived values afterwards, so anyone who wants different S3A settings for a catalog can still say so, and catalogs that declare no assume-role arn are untouched. `BaseProcedure` already holds the catalog (`tableCatalog()`), so the name is available where the action is built.

It is a behaviour change for catalogs that do declare a role: their listing starts going through that role rather than through the cluster identity. That's the point of the fix, and it matches what the rest of the procedure already does, but it is worth calling out.

I have this working against `spark/v4.1` with tests, and can open a PR plus the v4.0 and v3.5 backports. If the preference is to leave the Hadoop path alone because the FileIO one is where things are heading, then at minimum the docs should say that the listing uses the session's Hadoop config and not the catalog's, because nothing in the config surface hints at it.

For anyone else hitting the same 403, the workaround is to give S3A the same role, scoped to the buckets:

```
spark.hadoop.fs.s3a.bucket.my-lakehouse-bucket.aws.credentials.provider org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
spark.hadoop.fs.s3a.bucket.my-lakehouse-bucket.assumed.role.arn arn:aws:iam::111122223333:role/lakehouse-access
spark.hadoop.fs.s3a.assumed.role.credentials.provider com.amazonaws.auth.WebIdentityTokenCredentialsProvider
```

Per-bucket options key on the bucket name from the URI (`fs.s3a.bucket..`), independent of the scheme, so this covers `s3://` locations too. The role's trust policy has to allow the cluster principal to assume it, which it already does if the catalog is working.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Start in DeleteOrphanFilesSparkAction.java at the Hadoop configuration construction, then inspect SparkUtil.hadoopConfCatalogOverrides and BaseProcedure.tableCatalog() for the catalog context. Verify the catalog role settings and catalog-specific overrides are preserved, catalogs without an assume-role ARN remain unchanged, and the relevant Spark action or procedure tests cover the behavior.

Written by the indexing model from the issue text.