Epic: REST Catalog Scan Planning (server-side scan planning) client support

Bring iceberg-go to parity with Java and Python on REST server-side scan planning. The endpoints were added to the Iceberg REST OpenAPI spec by apache/iceberg#9695 (merged 2024-09-10) and shipped in Java through 2025 (apache/iceberg#13004, #13400, #14480, #14660, #14822, #15184, #15863, #16024, #16197).

Today iceberg-go has zero support: no types, no client methods, no scanner wiring, no capability discovery. `(*Scan).PlanFiles` reads manifests locally regardless of what the catalog can offer.

## What this unlocks

When this lands, Go consumers of an Iceberg REST catalog will be able to:

- Scan tables of any size without downloading manifests locally
- Run stateless from Lambda / Cloud Run / short-lived k8s pods (no manifest cache)
- Enforce catalog-side governance (RLS, masking, time-travel restrictions) on Go clients
- Receive per-scan vended `StorageCredentials` with plan-scoped TTLs
- Do cheap incremental scans for CDC consumers via `start-snapshot-id` / `end-snapshot-id`
- Drive the iceberg-go CLI against production-scale tables (`iceberg files`, `partition-stats`, `clean-orphan-files`, `expire-snapshots`, `compact`)

## Scope of this epic

This is a client-only effort. Non-goals:

- Implementing the server side of the spec (the REST catalog server lives in Java).
- A distributed worker runtime. We expose `FetchScanTasks` so external engines can fan out, but we don't build the engine.
- Reworking the local planning path. Default mode stays `local`; remote planning is strictly opt-in, no breaking change.

## Correctness contract

The acceptance test for the whole effort is the **local-vs-remote parity test**: the same scan (same filter, same table, same snapshot) under `scan-planning-mode=local` and `scan-planning-mode=remote` must produce the same `FileScanTask` set (paths, ranges, deletes, residuals). If it doesn't, the expression JSON codec is wrong and silent result corruption — not a loud error — is the failure mode. That codec is the highest-risk piece of the project.

## Proposed decomposition

Decomposable across PRs. The proposed breakdown below is **eight phases plus a parked follow-up**, sequenced so each lands behind a stable interface and ships independently. This is a starting point for the implementer to refine — please file sub-issues per phase as you pick up the work, adjusting scope and dependencies as the design firms up. The critical path is 0 → 1 → 2 → 3 → 4 → 5 → 6 → 7; Phase 2 (expression codec) can be developed in parallel with Phase 1.

- [ ] **Phase 0 — Capability discovery.** Extend `configResponse` in `catalog/rest` to decode `endpoints []string` from `GET /v1/config`; define endpoint constants matching the spec strings (`POST /v1/{prefix}/namespaces/{ns}/tables/{t}/plan` etc.); expose a typed capability check on `*rest.Catalog`. Pure additive, no behavior change. Good first issue. This is the discovery surface Phase 5's `auto` mode reads.

- [ ] **Phase 1 — Wire types.** Go types for the four planning payloads: `PlanTableScanRequest` (snapshot-id XOR start/end-snapshot-id, filter, select, case-sensitivity, use-snapshot-schema, metrics opts), `PlanTableScanResponse` (discriminated on `status`: completed/submitted), `FetchPlanningResultResponse` (completed/working/submitted/cancelled/failed), `FetchScanTasksRequest`/`Response`, `PlanStatus` enum, `PlanTask` opaque token, `CompletedPlanningResult` with `FileScanTask`s + optional `StorageCredentials`. Round-trip + golden JSON tests against fixtures from `rest-catalog-open-api.yaml`. XOR validation is the only logic.

- [ ] **Phase 2 — Expression JSON codec.** JSON codec for `BooleanExpression` matching the REST spec: `and`/`or`/`not`, all unary/binary/set predicates, column-name + nested-field-ID references, literal encoding for every primitive type (decimal w/ scale, timestamp vs timestamptz, UUID, fixed/var binary). Respect `case-sensitivity`. **Highest-risk phase** — cross-check against Java's `ExpressionParser` byte-for-byte. Get this wrong and filters silently differ between local and remote scans. No dependency on Phase 1, can run in parallel.

- [ ] **Phase 3 — REST client methods + poller.** `PlanTableScan`, `FetchPlanningResult`, `CancelPlanning`, `FetchScanTasks` on `*rest.Catalog`, wired via existing `doPost`/`doGet`. A `WaitForPlan` helper polling with jittered exponential backoff + configurable timeout (mirror Java apache/iceberg#15863). Error mapping: 404-during-poll → sentinel `ErrPlanExpired` so callers can re-plan (apache/iceberg#16024, #16197); 400/403 → typed errors; 503 → retry; ctx cancel → call `CancelPlanning` to free server resources, then return `context.Canceled`. Cancellation-on-cancel is the behavior that must match Java exactly — leaking plans on the server is vendor-visible.

- [ ] **Phase 4 — `ScanPlanner` interface + Table plumbing.** A `ScanPlanner` seam in `table/` so non-catalog code and tests can supply a planner without importing the REST package. `PlanResult` wraps inline `FileScanTask`s + opaque `PlanTask` tokens + plan-id (for cancel) + optional plan-scoped `StorageCredentials`. Thread the planner through `Table` (parallel to `CatalogIO`), wired when `rest.Catalog` loads the table; non-REST catalogs leave it nil. Public-API design moment — worth a design discussion in the PR. Open question: planner on `Table` vs handed into `Scan` at construction (recommend on `Table` so existing `t.Scan(...)` call sites don't change).

- [ ] **Phase 5 — Scanner delegation.** Add scan option `scan-planning-mode` with `local` (default), `remote` (require planner+capability, fail loud if absent), `auto` (remote if available, else local). Branch in `(*Scan).PlanFiles`; on remote, call `planner.PlanTableScan`, follow the sync/async branch via `WaitForPlan`, materialize `PlanTask` tokens via `FetchScanTasks` in batches, use plan-scoped `StorageCredentials` for subsequent FileIO. **The value-delivery moment** and where the correctness parity test lives. Small in code, huge in surface area.

- [x] **Phase 6 — Fake server + integration tests.** In-process fake at `catalog/rest/internal/planfake/` implementing the four endpoints over an in-memory table, running the same local planning code server-side (so parity tests are meaningful); supports sync/async/fanout/cancel via test switches. Plus an integration suite gated by `RUN_INTEGRATION_TESTS=1` against the Java `iceberg-rest-fixture` Docker image for interop confidence. Can be developed in parallel with Phases 4/5. _Landed via #1494 (scenario-driven client-conformance fake). Remaining Phase 6 scope spun out as follow-ups: local-vs-remote parity test (#1499) and Java `iceberg-rest-fixture` integration suite (#1500)._

- [ ] **Phase 7 — Hardening + docs.** Retries with jittered backoff, OpenTelemetry spans + metrics (plan requests, expirations, fallback-to-local), README "Plan Scan" row flipped to "Local + Remote", new "REST Scan Planning" section with capability matrix + config snippet, migration note (opt-in, no breaking change). Update v2 (#829) and v3 (#589) tracking issues. Closes the epic. Spin out any polish item that deserves its own follow-up rather than bundling.

- [ ] **Phase 8 (parked) — Distributed worker codec.** Public wire codec (`MarshalJSON`/`UnmarshalJSON` or protobuf) for `PlanTask` + `FileScanTask` so engines embedding iceberg-go can fan plan tokens out to workers. Only valuable to query engines that distribute plan-tasks; standalone Go consumers call `FetchScanTasks` inline (covered by Phase 5). Coordinate with apache/iceberg-go#1075 (`FileScanTask` codec for distributed compaction) to avoid duplicate work. Don't ship until an in-tree consumer needs it.

## Open questions to settle before Phase 1

1. **Scan option name.** Java settled on `RequiresRemoteScanPlanning` → `SupportsDistributedScanPlanning`. Pick a Go-idiomatic toggle name; match Spark wiring where sensible.
2. **Where `ScanPlanner` lives.** `table/` keeps it cohesive with `Scan` and lets tests inject a fake; `catalog/` puts it next to the catalog interfaces. Leaning `table/`.
3. **Default mode.** Start at `local` for back-compat; revisit `auto` once interop is proven against the Java fixture.
4. **Snapshot/schema semantics.** The spec lets clients pin `use-snapshot-schema`; make sure local-vs-remote parity covers schema-evolution edge cases.

## References

- Spec: apache/iceberg#9695 (merged 2024-09-10)
- Java umbrella + follow-ups: apache/iceberg#11180, #13004, #13400, #14480, #14660, #14822, #15184, #15863, #16024, #16197
- Related iceberg-go work: #1075 (FileScanTask wire codec), #829 (v2 tracking), #589 (v3 tracking)
- Overview: https://medium.com/data-engineering-with-dremio/iceberg-rest-catalog-overview-7-scan-planning-4b4bf2a46e4d