RFC: Add bq-assess — a read-only BigQuery→AWS lakehouse migration assessment CLI + Claude Code skill

## Summary

`bq-assess` is a read-only CLI that scans an existing **BigQuery** warehouse and produces a
migration assessment for moving it to an **AWS lakehouse** — data in **Amazon S3 Tables (Apache
Iceberg)**, queried by **Amazon Athena** or **Amazon Redshift Serverless**. It assesses both
engines by default and recommends a primary engine from the observed workload profile
(queries/day, bytes scanned, concurrency, latency SLA). It scores every table on two axes —
**Migration Effort** (how hard the data is to move) and **Query Complexity** (how hard the SQL
is to keep running on the target engine) — and generates Iceberg DDL in both Athena and
Redshift dialects, load guidance, and a BigQuery-vs-AWS cost comparison as HTML + JSON reports.
It ships both as a standalone CLI and as a **Claude Code skill** that guides users through
setup, the scan, and interpreting the results. It **assesses; it does not execute the
migration**, and needs no AWS account to run. It has been run against three customer BigQuery
environments during live migration engagements and five internal test environments.

> **Revision (July 2026):** the tool now lives in its own public repo —
> [aws-samples/sample-bigquery-to-aws-migration](https://github.com/aws-samples/sample-bigquery-to-aws-migration).
> This RFC no longer proposes vendoring code into `awslabs/startups` (original PR #139 closed as
> superseded). It now proposes a lightweight integration: a marketplace entry with an external
> source, plus a referral at `migration-to-aws`'s existing BigQuery specialist gate.

## Motivation

Customers evaluating a BigQuery→AWS move have no AWS-native way to see what the migration
will cost and how much work it will be before committing — one pilot customer spent weeks
manually migrating 1,000+ tables. Teams won't trust an automated migration without first seeing
a credible plan. `bq-assess` gives them that plan up front: a read-only scan (needs only
BigQuery metadata-viewer access, reads no data rows) that returns a cost delta, an engine
recommendation, and a per-table effort breakdown.

Today, `migration-to-aws` deliberately defers BigQuery: any `google_bigquery_*` resource is
marked **`Deferred — specialist engagement`** and the user is pointed at their AWS account team.
That policy is sound — IaC and billing data alone can't support a warehouse recommendation, and
"Athena or Redshift?" is exactly the question static inputs can't answer. `bq-assess` is that
specialist engagement in tool form: it derives the answer from the observed workload, and turns
the dead end into a runnable next step — the user walks into the account-team conversation
holding a data-backed report.

## Proposed Solution

No code lands in `awslabs/startups`. One small PR:

**1. Marketplace entry** in `.claude-plugin/marketplace.json`, sourced externally and pinned to
a release tag (tag to be cut on the public repo before the PR opens):

```json
{
"name": "bq-assess",
"source": { "source": "github", "repo": "aws-samples/sample-bigquery-to-aws-migration", "ref": "v0.4.0" },
"description": "Read-only BigQuery migration assessment — two-axis scoring (Migration Effort + Query Complexity), an Athena-vs-Redshift engine recommendation from the observed workload, Iceberg DDL, and a BigQuery-vs-AWS cost comparison. Assesses only; needs no AWS account to run."
}
```

`/plugin install bq-assess@startups-for-aws` then works for everyone who has already added this
marketplace.

**2. Referral at the existing BigQuery specialist gate** (no logic or policy change):

- `migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/clarify/clarify.md` —
the mandatory BigQuery advisory additionally offers a data-backed assessment via `bq-assess`:
the agent prints the install command, and if the plugin is already installed, offers to run it
in the same session (with user consent).
- `migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/design/design-infra.md` —
the specialist-gate text mentions the bq-assess report as an input to the account-team
conversation.

The `Deferred — specialist engagement` marker and the no-automated-target policy stay exactly
as they are: `migration-to-aws` still never names a warehouse target. The engine question
(Athena vs Redshift) is answered by `bq-assess` as the specialist tool, from observed workload
data — the division of labor the gate already prescribes, with a tool on the other end. With
both plugins installed the experience is one conversation: BigQuery detected → advisory → user
consents → the bq-assess skill runs inline (preflight → scan → interpret) → the
`migration-to-aws` flow resumes.

Both tools stay independent: separate repos, CI, and release cadence. The marketplace pins a
tag; upgrades are a deliberate one-line `ref` bump by maintainers.

## Alternatives Considered

- **Vendor the code into `awslabs/startups`** (original proposal; PR #139) — rejected:
duplicate source of truth, drift between copies, and this repo taking on a Python CLI's
CI/test burden. Superseded by the external-source marketplace entry.
- **Deep integration into the `gcp-to-aws` skill** — rejected for v1: the skill's inputs are
static files (Terraform/billing/app code) while `bq-assess` needs live read-only GCP
credentials; phases run under a strict context budget; and the specialist-gate policy is a
deliberate product decision. A structured handoff is listed as optional follow-up under Open
Questions.
- **Keep it internal-only** — rejected; the value is a public, self-serve tool customers and
partners can run themselves.
- **AWS Schema Conversion Tool (SCT)** — complementary rather than competing. SCT focuses on
schema and code conversion; `bq-assess` is assessment-first, adding a BigQuery-vs-AWS cost
comparison, an Athena-vs-Redshift engine recommendation, and effort scoring tuned for
complex/nested schemas mapping onto S3 Tables/Iceberg.
- **Generic multi-source assessor** (Snowflake, Teradata, etc. as sources) — deferred; depth on
the BigQuery source first. On the AWS side the tool already assesses both Athena and Redshift.

## Open Questions

- ~~Preferred home: an artifact folder inside `awslabs/startups`, or a dedicated repo?~~
**Resolved: dedicated public repo** (`aws-samples/sample-bigquery-to-aws-migration`),
referenced from this marketplace as an external source.
- Version pinning cadence: we cut release tags on our side; maintainers bump the pinned `ref`
when they choose. Does that workflow suit the team?
- **Optional phase 2 (future, needs maintainer buy-in):** a structured handoff where the
`gcp-to-aws` skill ingests bq-assess's JSON report (cost delta, engine recommendation,
per-table effort counts) into `migration-report.html` instead of the bare `Deferred` rows.
Requires a JSON schema contract and a policy decision on the gate — proposed as follow-up,
not part of this RFC's PR.

## Drawbacks

- Scope is BigQuery as the only source for v1 — not a general multi-warehouse assessor (on the
AWS side it covers both Athena and Redshift).
- Confidence depends on inputs: without query logs, some scores, the engine recommendation, and
the cost estimate are clearly-labelled ranges rather than exact figures.
- Phase 1 produces two linked reports: `migration-to-aws`'s report keeps its `Deferred` rows,
and the bq-assess report is a separate artifact. Merging them is the optional phase-2
follow-up above.
- The scan requires live GCP credentials (ADC + `roles/bigquery.metadataViewer`), unlike the
static-file inputs the skill otherwise uses — surfaced clearly in the advisory and in the
skill's preflight phase.