aws / aws/aws-durable-execution-sdk-python

Improve the monorepo release process

Abierto
#716 1 comentario 0 reacciones 0 asignados Ver en GitHub
enhancement pkg:otel pkg:sdk pkg:testing
Lenguaje dominante
Python
Estrellas
53
Forks
25
Merge medio
1 d 19 h
PR fusionados (30 d)
40

Descripción

### Problem

Python has no native tooling for releasing multiple packages from one monorepo. There is no equivalent of npm workspaces with per-package publish orchestration. This repository works around the gap with a GitHub Actions workflow, `pypi-publish.yml`. The workflow builds and publishes every package on every GitHub release. This design has three problems.

**1. The release tag does not control what gets published.**

The workflow triggers on `release: published`. It uses the tag only as a checkout ref. No filter connects the tag name to the package matrix. So a release tagged `sdk-v2.0.0` builds and uploads all four packages: `aws-durable-execution-sdk-python`, `-otel`, `-testing`, and `-insight`.

The real selector is each package's `__about__.py` version. A package publishes when its version is absent from PyPI. A package fails when its version is already there. So the set of published packages depends on which version bumps sit on the tagged commit. It does not depend on what the tag says. A maintainer who intends to release only the SDK can unintentionally publish other packages.

One workflow does read the tag. `lambda-layer-publish.yml` gates on `contains(tag_name, 'otel-v')`. That gate forced the combined tag convention `sdk-v1.7.0,otel-v0.3.0`. The convention packs several package names into one git tag with commas.

**2. Already-published versions fail the workflow instead of being skipped.**

The publish step does not check PyPI before uploading. It does not set `skip-existing`. A package with an unchanged version is rebuilt and re-uploaded. PyPI rejects the upload with `400: Uploading new files to releases older than 14 days is not allowed`. In practice the `-testing` package hits this on most releases. So every release since the monorepo conversion has finished with a failed workflow run. This holds even when the intended packages published successfully. A release workflow that is always red trains maintainers to ignore failures. Ignored failures hide real ones.

**3. There is no pre-flight plan and no post-publish verification.**

Nothing reports which packages will publish before uploads begin. Nothing confirms afterward that PyPI serves the new versions. The first signal of a wrong or missing publish is a user report, or a red job found by reading logs.

### How the JS repository avoids this

[aws-durable-execution-sdk-js](https://github.com/aws/aws-durable-execution-sdk-js) has the same outer shape. A release event triggers a workflow that iterates all publishable packages. The difference is its publish script, `iterate-publish-npm.sh`. The script runs in two phases.

Phase one is a pre-flight. Before any upload, the script reads the npm registry state for every package. It decides a plan per package: publish under a computed dist-tag, skip because the version is already published, or reject for a policy violation. A reject aborts the run before any upload. An unreadable registry also aborts the run.

Phase two publishes according to the plan. A skip is a deliberate decision and the run stays green. After each upload the script polls the registry until the dist-tag points at the published version.

This makes releases idempotent. A release evaluates six packages and publishes only the ones whose versions moved. Single-package releases such as `sdk-2.3.1` and combined releases such as `sdk-2.4.0/test-1.1.4/otel-1.0.0` go through the same workflow.

The JS model still does not make the tag authoritative. Package selection there is also version-file-driven. The proposal below goes one step further.

### Proposal

1. **Make the tag the source of truth for package selection.** Define one tag grammar. For example, `sdk-v2.0.0` names a single package and `sdk-v2.0.0+otel-v1.0.0` names a combined release. A preflight script parses the tag. Only the named packages are built and published. If a tagged version disagrees with the package's `__about__.py`, the preflight fails before any upload.
2. **Add a preflight plan job.** For each selected package, compare `__about__.py` against the PyPI JSON API. Emit a plan of publish, skip, or error to the job summary. Downstream jobs consume the plan.
3. **Make publishing idempotent.** Set `skip-existing: true` on `pypa/gh-action-pypi-publish`. A re-run of a partially failed release then republishes only what is missing.
4. **Verify after publishing.** Poll the PyPI JSON API until the new version is served. Bound the retries.
5. **Apply the same tag gate to the downstream release workflows.** These are the Lambda layer and ECR emulator image workflows. An SDK-only tag must not rebuild or retag unrelated artifacts.

### Acceptance criteria

1. **No tag/publish discrepancy.** A release tagged for package X publishes exactly X. Publishing a package not named in the tag is impossible. This holds regardless of which version bumps exist on the tagged commit.
2. **Protection against accidental and inconsistent publishes.** The preflight aborts before any upload when a tagged version disagrees with the source version. It also aborts when a policy check fails. Nothing is published on abort.
3. **Successful workflow runs on every legitimate release.** An already-published version is a reported skip, not a failure. A green run means everything intended was published and verified. A red run means something went wrong. Re-running a release is safe and idempotent.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Empieza leyendo .github/workflows/pypi-publish.yml y el archivo relacionado lambda-layer-publish.yml, junto con los workflows de imágenes del emulador de ECR. Traza cómo interactúan actualmente las etiquetas de release, las versiones de los paquetes de __about__.py y los pasos de publicación; después, compara el flujo de preflight y verificación propuesto con el modelo del repositorio JS de referencia. Se considera terminado cuando los paquetes seleccionados por etiqueta se publican de forma segura, las discrepancias interrumpen el proceso antes de la carga, las versiones ya publicadas se omiten y los releases exitosos se verifican.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
github-actions, python
Área
ci-cd, release
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.