Per-user BYO-S3 warehouses

## Background

Every execution's outputs — results, runtime statistics, and console logs — are currently written to a single shared Iceberg warehouse, with the platform absorbing the S3 cost. There is no per-tenant storage isolation — one warehouse holds every user's tables — and no way to attribute storage cost to the user who owns the data. Design discussion: #5293.

## Goal

Let each user register a *warehouse* — a Lakekeeper catalog entity with its own storage configuration and key-prefix — and let a workflow execution write its Iceberg tables (results / runtime statistics / console messages) into the warehouse it selects. Two motivations, and only the first requires the user to bring their own bucket:

- **Cost attribution (Cloud flavor).** With the user's own S3 bucket, storage cost lands on the data owner's bill, and users who bring their own storage keep full control of their data.
- **Per-tenant storage isolation (both flavors).** Each user's tables live in their own warehouse, under their own key-prefix, instead of mixed into one shared warehouse — user data gets an owner and a boundary. Lakekeeper attaches the storage profile and credential at warehouse granularity, which Phase 1 builds on. This holds on the deployment's own MinIO just as on a user-owned bucket, so warehouses are useful even where nobody brings their own storage.

## Architecture

## Design

**Warehouse (Design 2, #5293).** A warehouse is a Lakekeeper (Iceberg REST) catalog entity backed by the user's S3 bucket, holding the execution namespaces (results, runtime statistics, console messages) plus its storage configuration. Each warehouse is a distinct Lakekeeper warehouse with its own key-prefix in the bucket. It is selected **per execution**, decoupled from the computing unit.

**Two-layer runtime.**
- *Catalog path* — a per-warehouse REST catalog client talks to Lakekeeper for metadata.
- *Data path* — Iceberg reads/writes Parquet **directly** to the user's S3 using Lakekeeper-vended short-lived credentials, **continuously refreshed** for the life of the execution (see below).

**Credentials: assume-role, no static keys (#6040).** The user grants Texera an IAM role; the **platform principal** (this deployment's AWS identity, provisioned once by the operator) assumes it via AWS STS, gated by a **per-warehouse external ID**; workers receive only short-lived STS credentials, **continuously refreshed** — not vended once per execution. The external ID is **server-minted** by the Texera web service (the client cannot choose it — this prevents registering another user's role ARN + external ID to reach their bucket), **per-warehouse**, and **unguessable (UUID)**; it is reserved **before** create so the user wires their role's trust policy in one step (no reveal round-trip). Texera stores only **non-secret metadata** — the Lakekeeper warehouse id, role ARN, and external ID — never raw S3 keys. Assuming a role requires all three at once (platform credentials + external ID + role ARN), so a leaked role ARN + external ID grant nothing on their own.

**Credential lifetime — a precondition, not an assumption.** The data path uses vended credentials — the model managed Iceberg REST catalogs have converged on — so workers hold short-lived STS credentials (1h by default, and capped at 1h whenever Lakekeeper's own identity is itself an assumed role, e.g. IRSA) while an execution can run much longer. An Iceberg writer resolves its table — and therefore its `FileIO` — once, at construction; `table.refresh()` updates metadata but **never replaces the `FileIO`**, so a writer cannot pick up new credentials on its own. The standard answer, and ours, is client-side refresh: Lakekeeper must return `client.refresh-credentials-endpoint` in the table config, which lets Iceberg's `VendedCredentialsProvider` renew on its own. Without it Iceberg silently falls back to a static provider, and every S3 write after expiry fails with no retry path (for runtime statistics that failure is swallowed and only logged). Warehouse registration should probe for this endpoint and refuse to create the warehouse when it is absent.

**Warehouse types (flavor).** Both flavors are warehouses in the same sense — a catalog entity with its own storage profile, credential and key-prefix — and differ only in who owns the bucket and how it is reached. *Local* is backed by the deployment's own object store (MinIO) with the platform's static credentials, needs no user setup, and is the flavor a single-tenant or on-prem deployment would use exclusively. *Cloud* (AWS) points a warehouse at the user's own bucket via the assume-role flow below, which additionally moves the storage bill to the data owner.

## Warehouse setup (user side, Cloud/AWS)

When the user opens the Cloud form, Texera shows the **platform principal ARN** and a reserved **external ID**. The user creates one IAM role with:

**Trust policy** — allow the platform principal to assume the role, gated by the external ID:
```json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "AWS": "" },
"Action": "sts:AssumeRole",
"Condition": { "StringEquals": { "sts:ExternalId": "" } }
}]
}
```

**Permission policy** — grant the role access to the bucket:
```json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::", "arn:aws:s3:::/*"]
}]
}
```

## Warehouse lifecycle

- **Create** — reserve identity (external ID + platform principal ARN) → user creates the IAM role (policies above) → submit bucket / region / role ARN → Texera (via Lakekeeper) validates by **actually assuming the role and test-writing to S3**. A wrong role ARN / external ID / bucket fails **at create time** (HTTP 502), before any record is written — no orphaned state.
- **Delete** — the warehouse is emptied in the catalog first (Lakekeeper refuses to drop a non-empty warehouse): its tables and namespaces are dropped, and the underlying **S3 data files are purged along with them**. This matches how Texera already deletes execution results today (`IcebergDocument.clear()`, and table re-creation on every re-run), and it avoids leaving the user with orphaned Parquet and metadata files that no longer map back to anything. Two consequences to be explicit about:
- **Scope.** A purge only removes the objects Iceberg itself recorded for those tables — i.e. what Texera wrote under this warehouse's key-prefix. Nothing else in the user's bucket is touched.
- **It is destructive and it happens in the user's own bucket**, so the UI must say so and require confirmation before deleting a warehouse. This is also why the permission policy above keeps `s3:DeleteObject`.

## Implementation surface

- **Backend** — `WarehouseResource` (REST) · `LakekeeperClient` (warehouse management + empty-first delete) · `StorageConfig` (the `storage.warehouse.enabled` flag) · `WorkflowService` (resolves the per-execution warehouse, ownership-checked) · per-warehouse `IcebergCatalogInstance` cache · warehouse-scoped storage URIs (`DocumentFactory` / `VFSURIFactory`).
- **Schema** — `user_warehouse`, `user_s3_pending_identity`.
- **Frontend** — Warehouse tab, on-canvas per-execution picker, AWS IAM-role warehouse form, warehouse service.

## Scope & rollout

- **Target: REST-catalog (Lakekeeper) deployments.** Single-node / local Docker Compose on the Postgres catalog is unchanged — it keeps its single shared warehouse.
- Ships behind a **default-off** flag (`storage.warehouse.enabled`). With the flag off, storage behaves exactly as it does today.

The credential model rolls out in three phases, planned in #6040. Those phases all assume a user-owned bucket; the warehouse *concept* does not, so it lands first:

**Phase 0 — warehouses on the deployment's own storage (Local flavor, static keys).** Per-user warehouses over the deployment's own MinIO: per-tenant isolation, and a per-warehouse owner to account storage against. No STS, no assume-role, no external ID, no credential refresh — none of that is on the critical path — and it is fully testable in CI, which already provisions Lakekeeper + MinIO. **A deployment can turn the flag on at the end of Phase 0.**

**Phase 1 — assume the user's role (Cloud / BYO-S3).** #6040 Phase 1. From here on an execution writes into the user's own AWS account. It is also the phase that is effectively untestable in CI, since assume-role needs a real AWS identity.

**Phase 2 — authenticate the Computing Unit.** #6040 Phase 2. The CU authenticates to Lakekeeper (OIDC) so vended credentials reach only the execution they were minted for. Hard prerequisite for enabling Phase 1 in a multi-tenant deployment.

**Phase 3 — keyless deployment.** #6040 Phase 3. Provision the per-deployment system identity via IRSA / instance profile and wire it through Terraform.

The sub-issues below cover Phases 0 and 1; Phases 2-3 stay in #6040 until they are picked up.

**Two gates, not one.** `storage.warehouse.enabled` turns on the warehouse concept and can be enabled once Phase 0 lands; a separate flag gates the Cloud/BYO flavor and stays off until Phase 2, so enabling warehouses never turns on the BYO path early. `storage.warehouse.enabled` is a rollout switch: it can be retired — together with the shared-warehouse path — once every user is provisioned a warehouse automatically.

Note that all of this applies to REST-catalog (Lakekeeper) deployments only. The postgres and hadoop catalogs have no warehouse entity — their `warehouse` property is just a single configured root location — so a single-node deployment is unaffected and gains nothing here.

## Sub-issues (each → one PR)

**Phase 0 — warehouses on the deployment's own storage**

- [x] Storage foundation: per-warehouse Iceberg catalog + warehouse-scoped storage URIs (backward-compatible no-op)
- [x] Feature flag (storage.warehouse.enabled, default off)
- [x] Schema: `user_warehouse`
- [x] Warehouse REST endpoints + Lakekeeper client + per-execution injection
- [ ] Frontend: warehouse dashboard tab — #6933
- [ ] Frontend: on-canvas per-execution warehouse picker — #7817
- [x] Deployment: give the default warehouse its own key prefix so per-user warehouses can be created — #7739
- [x] Bound the per-warehouse catalog cache and release what it evicts — #7290
- [x] Warehouse deletion: wait out Lakekeeper's asynchronous purge instead of failing with 409 — #7742
- [x] Expose the warehouse owner in `DashboardWarehouse` so the UI can bind to it — #7743
- [ ] Require a warehouse for every execution while the feature is enabled — #7751
- [x] Decouple the Lakekeeper warehouse name from the user-facing name — #7753

**Phase 1 — assume the user's role (BYO-S3)**

- [ ] Separate flag for the Cloud/BYO flavor, off until Phase 2: `storage.warehouse.enabled` must not be what exposes user-owned buckets, or enabling Phase 0 would also enable a path that needs CU authentication
- [ ] Schema: assume-role fields (role ARN / external ID / pending identity)
- [ ] Backend: assume-role warehouse creation + reserve-identity
- [ ] Frontend: AWS IAM-role warehouse form

## Follow-ups (out of scope here)

- **Per-warehouse usage accounting** — attribute stored bytes to the warehouse holding them (`UserQuotaResource` sums per `uid` only), so Local-flavor storage has an owner to charge back.
- **Warehouse sharing** — per-warehouse access control (ACL) for collaboration.

### Affected Area

Storage / Metadata