TypeScript SDK: embed source and integrity sections in packed bundles
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Background
#69295 adds `airflow-ts-pack`, which embeds the bundle manifest as a single first-line comment in `bundle.mjs` (`//# airflowMetadata=`); the Node coordinator reads only that head line (bounded `readline()`, 1 MiB cap).
The Go SDK's AFBNDL01 executable bundle format provides two things the TS format does not (see the Trailer Layout section of `task-sdk/docs/executable-bundle-spec.rst`):
- **Integrity validation** — a SHA-256 digest over the code region, verified by the scanner before the bundle is used.
- **An embedded source region** — the original entrypoint (`.ts` / `.js`), so the UI can display the code; needed for the native TypeScript Dag feature (#69288).
Review on #69295 agreed to keep that PR to the packing flow and refine the packing spec in a follow-up. Since `bundle.mjs` must remain directly runnable by `node`, the structure cannot be a binary EOF trailer like AFBNDL01; instead the head comment can carry the layout — a format version, start/end offsets for each section, and the digest.
### What needs to happen
1. Specify a versioned head-comment layout for `.mjs` bundles recording section offsets (bundled code, embedded source, metadata) and a SHA-256 digest of the code region, documented alongside AFBNDL01 in `task-sdk/docs/executable-bundle-spec.rst`.
2. Update `airflow-ts-pack` (`ts-sdk/src/cli/pack.ts`) to emit the new layout and embed the original entrypoint source.
3. Update the Node coordinator (`task-sdk/src/airflow/sdk/coordinators/node/coordinator.py`) to verify the digest during scan — log and skip on mismatch, with the same `(path, inode, mtime, size)` result caching as the AFBNDL01 reader algorithm — and expose the embedded source.
### Acceptance criteria
- A packed `bundle.mjs` remains directly runnable with `node bundle.mjs`.
- The coordinator logs and skips a truncated or modified bundle on digest mismatch.
- The embedded entrypoint source is retrievable for display.
- `node-bundle-spec.rst` documents the TS / JS layout with its own format version.
### Context
- Originating PR: #69295
- Review threads: metadata placement (trailer moved to head): https://github.com/apache/airflow/pull/69295#discussion_r3520504501 ; deferred spec refinement ("we can refine the packing spec in the follow-up"): https://github.com/apache/airflow/pull/69295#discussion_r3551043871
- Go parallel: AFBNDL01 trailer layout, `task-sdk/docs/executable-bundle-spec.rst`
- Related: #69288 (TypeScript SDK: native TypeScript Dag declaration — the source-viewing consumer)
Contributor guide
Research direction
Start with the Trailer Layout section in task-sdk/docs/executable-bundle-spec.rst, then inspect ts-sdk/src/cli/pack.ts and task-sdk/src/airflow/sdk/coordinators/node/coordinator.py alongside the AFBNDL01 reader algorithm. Define the versioned head-comment layout, packing and scan behavior from the acceptance criteria; done means runnable bundles, digest mismatch logging and skipping, and retrievable embedded source, with node-bundle-spec.rst updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, python, typescript
- Domain
- backend, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100