apache / apache/texera

[BYO-S3] Backend: assume-role warehouse creation + reserve-identity

Open
#6,935 0 comments 0 reactions 1 assignee Claimed by @mengw15 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Part of #6870. The backend for the AWS path, where the user grants an IAM **role** instead of
handing over long-lived access keys.

- **Reserve an identity up front** — a `POST /warehouse/reserve-identity` endpoint: the web service
mints a per-warehouse, server-generated `external_id` (a random UUID; reuses the existing pending
one if present) and returns it with the deployment's platform principal ARN, so the user can wire
their IAM role's trust policy in one step, before the warehouse exists.
- **Assume-role create** — creating an AWS-flavor warehouse binds the reserved external ID and, via
`LakekeeperClient`, registers a Lakekeeper warehouse whose **storage profile** carries
`assume-role-arn` + `sts-enabled`, and whose **storage credential** is `aws-system-identity` with
the `external-id`. Lakekeeper validates by **actually assuming the role and test-writing to S3**,
so a wrong role ARN or external ID fails at create time (HTTP 502), before anything is saved.
- **Refuse a warehouse whose credentials cannot refresh** — vended STS credentials are short-lived
(1h by default, and capped at 1h whenever Lakekeeper's own identity is itself an assumed role),
while an execution can run much longer. Iceberg renews them only when the table config carries
`client.refresh-credentials-endpoint`; without it the client silently falls back to a static
provider and every S3 write after expiry fails with no retry path — for runtime statistics that
failure is swallowed and only logged. Probe the created warehouse's table config for the endpoint
at registration time and refuse the warehouse when it is absent, instead of handing back one that
works for the first hour and then stops.
- **Platform identity config** — a `platform-role-arn` setting (`StorageConfig` / `storage.conf`):
the deployment's AWS identity ARN, shown to users so they know whom to trust in their role;
empty when the deployment has no AWS identity.

### Task Type
- [x] Other (feature implementation)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.