New community transformation: glue-version-upgrade-readiness (AWS Glue 2.0/3.0/4.0 → 5.0 readiness)

# Proposal: Add AWS Glue 5.0 Upgrade Readiness Transformation

## Problem

AWS Glue 5.0 upgrades the managed ETL runtime to Spark 3.5.4, Python 3.11, Scala 2.12, and Java 17. Teams upgrading from Glue 2.0, 3.0, or 4.0 need to review Spark behavior and deprecated APIs, Python dependencies, Glue DynamicFrame and logging behavior, JDBC/custom connectors, native data lake formats, and the Terraform/CloudFormation/CDK definitions that select the runtime.

The work is error-prone because some changes are mechanical while others require data-level regression tests or vendor/environment evidence. In particular, custom Marketplace connectors, compiled dependencies, removed Spark 2.x behavior, and cross-account Data Catalog/Lake Formation cases should not be guessed.

## Proposed Scope

Add a community-sourced transformation named `glue-version-upgrade-readiness` that:

- detects PySpark and Scala Glue jobs plus Terraform, CloudFormation, and CDK definitions;
- infers or reads the source Glue version and uses a conditional Phase 0 pipeline;
- prepares Glue 2.0/3.0/4.0 jobs for Glue 5.0;
- handles Spark 2.4/3.1/3.3 → 3.5, Python 3.7/3.10 → 3.11, Glue API/logging, connectors/formats, and IaC updates;
- keeps business logic, job names/arguments, output schemas, and tests unchanged;
- treats Marketplace connectors, native dependencies, non-mechanical Spark behavior, and cross-account catalog cases as flag-only;
- generates `MIGRATION_REPORT.md` for every run;
- requires already-compatible control files to remain byte-identical.

The skill prepares repository code only. It will not deploy jobs, call AWS APIs to update jobs, run production workloads, or change data.

## Proposed Structure

Mirror existing community skills:

```text
community-sourced-transformations/glue-version-upgrade-readiness/
├── README.md
├── SKILL.md
├── BENCHMARKS.md
└── references/
├── spark2-to-spark35-api-changes.md
├── glue-api-and-dynamicframe-changes.md
├── python311-compatibility.md
├── connectors-and-datalake-formats.md
├── iac-glue-job-updates.md
├── verify-legacy-patterns.md
└── examples-before-after.md
```

Technical claims are grounded in the official AWS Glue 5.0 migration guide/release notes, Apache Spark 3.5 migration guides, Python 3.11 documentation, and official IaC references. Where primary documentation does not define a mechanical transform—general DynamicFrame deprecation, encoder-dependent Scala Dataset behavior, or cross-version CDK alpha compatibility—the skill states a known limitation and requires manual review.

## Validation Plan

Benchmark three seeded repositories:

1. PySpark Glue 2.0 job with Spark/Python/logging incompatibilities;
2. Scala Glue 3.0 job with Scala 2.11, deprecated view API, and an untyped UDF;
3. Terraform-managed Glue 4.0 Iceberg job with obsolete logging arguments and a flag-only Marketplace connector.

For each run, compare against a git baseline, apply the skill's 20 exit criteria and blocking grep sweeps, run applicable Python/Scala/IaC validators, verify `MIGRATION_REPORT.md`, and confirm a planted compatible control remains byte-identical.

## Validation

The local `atx` executable was unavailable, so the three committed fixtures were transformed manually phase-by-phase and are labeled **manually simulated, not agent-executed** in `BENCHMARKS.md`.

- Python 3.11.15 compiled all fixture scripts and all 10 extracted Python fenced examples.
- Terraform 1.15.8 with AWS provider 5.100.0 passed formatting and validation for both Terraform fixtures and two extracted IaC examples.
- cfn-lint 1.53.3 passed all four extracted CloudFormation examples after they were wrapped in complete validation templates.
- sbt 2.0.4 compiled the exact transformed Scala sources against Scala 2.12.18/Spark 3.5.4 in a minimal GlueContext harness. The full seeded build correctly remains failed because its deliberately unpublished connector is flag-only.
- All blocking sweeps ran; 14/14 planted findings were detected, 10/10 mechanical changes were correct, 4/4 manual items remained unmodified and reported, and all three controls were byte-identical.
- Skill frontmatter, Markdown links, JSON, whitespace, TODO-marker absence, and second-pass idempotence were checked.

Production Glue runs and live AWS Transform Custom benchmarks remain outside this local validation.

Start by reviewing existing community-sourced transformations, then create the proposed community-sourced-transformations/glue-version-upgrade-readiness/README.md, SKILL.md, BENCHMARKS.md, and references files. Use the three seeded repositories and the listed validation tools as the first validation pass. Done means the 20 exit criteria, blocking sweeps, reports, idempotence checks, and byte-identical controls all pass, with manual items remaining flagged.

Written by the indexing model from the issue text.