Publish a Python client + types for the Project Engine API (transport parity with the TS client)
- Dominant language
- JavaScript
- Stars
- 5
- Forks
- 2
- Avg merge
- 19h 55m
- Merged PRs (30d)
- 33
Description
## Summary
Provide a **published, CI-verified Python client + types** for the Project Engine API in `@adobe/spacecat-shared-project-engine-client`, at parity with the existing TypeScript client's transport behaviour. Today the package generates Pydantic **models only** — there is no Python transport (auth, retry, base-URL handling), the generated Python is not published so no other repo can depend on it, and it is not regenerated or drift-checked in CI. As a result Python consumers hand-roll the whole client, and at least one such hand-rolled client has a real correctness gap the shared TS transport already avoids.
## What exists today (v1.10.0)
The generation pipeline lives in `packages/spacecat-shared-project-engine-client`:
- Source spec: `spec/projectengine_swagger_public.yaml` (Swagger 2.0, "Project Engine API" v1.0) → `swagger2openapi` → `build/openapi3.json` → overlay `spec/overlays/corrections.yaml`.
- `generate:ts` → `src/generated/types.ts` (openapi-typescript). The **TS client** (`src/client.js`, ~135 lines, plus `src/internal.js`) is a thin `openapi-fetch` wrapper over those types that OWNS the transport: auth middleware, method-aware retry, base-URL normalisation.
- `generate:pydantic` → `python/serenity_project_engine/` (`model.py`, `http_server.py`, `aiseo.py`, `__init__.py`) via `datamodel-codegen` — **Pydantic v2 models ONLY**.
## The gap
1. **No Python transport client.** `datamodel-codegen` emits data models, not a client. There is no Python equivalent of `src/client.js` — no Bearer-token auth injection, no retry/backoff, no base-URL handling. A Python consumer gets typed request/response shapes but must write and maintain all the HTTP plumbing itself.
2. **The Python artifacts are not consumable.** `package.json` `files: ['src']` excludes `python/`, and there is no PyPI publish or git-branch distribution for it. So the committed Pydantic models cannot be installed or `git+`-depended-on by another repo today. (Contrast: `mysticat-data-service` distributes its own generated Python types via a `types-py` orphan git branch — a proven pattern in this org.)
3. **No CI regeneration / drift-check.** `generate:pydantic` requires Python and is deliberately kept manual (see the comments in `.github/workflows/project-engine-mock-e2e.yaml` and `.github/workflows/user-manager-mock-e2e.yaml`). The committed Python surface can therefore silently lag the spec and the TS types, with nothing to catch it — whereas `generate:ts` is verified in CI.
## Motivating consumer (and a concrete defect it demonstrates)
`mysticat-data-service` `scripts/serenity_migration/semrush_write.py` defines `SemrushWriteClient`, a bespoke `httpx` client against this exact API (`/enterprise/projects/api/...`, IMS Bearer). It re-implements auth, retry/backoff, pagination, and the wire shapes by hand.
Its retry is **not method-aware**: `_RETRY_STATUSES = {429, 502, 503, 504}` are retried for **every** verb, including `POST`, with no `Retry-After` handling. A `POST` that returns 502 *after* the server already applied it gets replayed → a potential duplicate write (e.g. `create_project`, `create_benchmarks`). The TS transport avoids exactly this by retrying 5xx/network errors only for idempotent methods (see `src/internal.js`). A shared Python client would give every Python consumer that correctness for free instead of each re-deriving it (often wrong).
The dimension-root tag migration program is the immediate driver; this migration CLI is a **living** tool (`--reconcile` is used for ongoing customer onboarding), so removing its hand-rolled transport layer has lasting value, not one-shot.
## Scope / tasks
- Add a thin Python **transport client** (over `httpx`) in this package that consumes the generated Pydantic models and replicates the TS client's cross-cutting behaviour (see "Transport contract" below).
- Decide and implement a **distribution mechanism** for the Python package (PyPI, or a `types-py`-style orphan git branch, or a `git+https` subdirectory dependency) and stop excluding it from what ships. Document how consumers depend on it.
- Wire `generate:pydantic` (and the new client) into **CI as a drift-check** — regenerate and fail on diff, mirroring how the TS surface is guarded — so the committed Python can't lag the spec.
- Provide minimal tests for the transport client (auth injection, method-aware retry incl. `Retry-After`, base-URL normalisation) and, ideally, an e2e smoke against the existing stateful mock (`mock/`, served under `/enterprise/projects/api`).
- **Acceptance / first consumer:** `mysticat-data-service` `SemrushWriteClient` is re-backed onto (or replaced by) the shared Python client for the Project Engine surface, dropping its hand-rolled wire + retry layer.
## Transport contract the Python client must replicate
From the TS client (`src/client.js`, `src/internal.js`):
- **Base URL:** hard-code `API_PREFIX = /enterprise/projects/api`; from the caller's `baseUrl` keep only `protocol//host` (drop any path/credentials); reject non-http(s).
- **Auth:** `Authorization: Bearer `; token source is a value or a callable (sync/async), resolved once per logical request and reused across that request's retries; fail fast on empty/invalid source. The raw IMS JWT is forwarded verbatim — no token exchange/minting.
- **Retry:** retry `429` for **all** methods; retry `5xx` and network errors only for **idempotent** methods (`GET/HEAD/PUT/DELETE/OPTIONS`). Equal-jitter exponential backoff (`base * 2**attempt * rand[0.5,1)`), honour `Retry-After` (seconds or HTTP-date), clamp to 20000 ms. Defaults: maxRetries 2, base delay 200 ms. Replay bodied requests safely (per-attempt body handling). Best-effort `onRetry` hook.
- **Model nuances:** tag **create** vs **read** use different shapes (`TreeNode*` request vs `AIOTag` response); `AIOTag.path` is an ordered array of `{id, name, parent_id}` leaves in **snake_case** (not a string, no client-side `parentId`/`dimension` invention); prompts/tags/benchmarks/brand_urls live on **v2 `/aio/...`**, with some deprecated v1 equivalents.
- **Error model:** the TS client does not throw on HTTP errors (returns `{data, error}`). The Python client should pick a deliberate, documented error model (the existing `SemrushWriteClient` raises a typed `SemrushError` — a reasonable Pythonic choice; just make it intentional).
## Explicitly out of scope (separate follow-ups, noted so no one conflates them)
- **The dimension-root DOMAIN model is NOT in this package** and this issue does not add it. Neither the TS nor the Python client encodes `dimension = path[0]`, `DIMENSION_ROOTS`, the closed `intent`/`source`/`type` taxonomies, or `buildTagsOf` — those are re-implemented independently in `spacecat-api-service`, `project-elmo-ui`, and `mysticat-data-service` (`scripts/serenity_migration/tags.py`). Generating a Python client does **not** dedupe that taxonomy. A single source of truth for the taxonomy is a distinct, higher-value follow-up and should be tracked separately.
- **Workspace provisioning is a different API.** The migration CLI also drives workspace create/status/transfer/delete/rename on `/enterprise/users/api/...`, which belongs to `@adobe/spacecat-shared-user-manager-client` (which has its own `generate:pydantic` in the same state). A full de-hand-rolling of the migration CLI needs the equivalent Python client there too — companion work, same pattern.
## Pointers
- Package: `packages/spacecat-shared-project-engine-client` (`@adobe/spacecat-shared-project-engine-client`, v1.10.0).
- Generation: `package.json` scripts `spec:convert` / `spec:overlay` / `generate:ts` / `generate:pydantic` / `generate`.
- TS transport to mirror: `src/client.js`, `src/internal.js`.
- Stateful mock for e2e: `mock/` (Counterfact, `/enterprise/projects/api` prefix).
- Consumer to migrate: `mysticat-data-service` → `scripts/serenity_migration/semrush_write.py` (`SemrushWriteClient`).
- Companion package: `@adobe/spacecat-shared-user-manager-client` (same Python-client gap, for the workspace/users surface).
Contributor guide
Research direction
Start by reading packages/spacecat-shared-project-engine-client/src/client.js, src/internal.js, package.json, and the generate scripts to map the existing transport and generated surfaces. Run the package generation and existing mock/e2e workflows first; done means a distributed Python client, transport tests for auth, retries, and base URLs, CI drift protection, and migration of mysticat-data-service/scripts/serenity_migration/semrush_write.py.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, javascript, openapi, python
- Domain
- api, backend, build-system, ci-cd, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100