apache / apache/airflow

TypeScript SDK: embed source and integrity sections in packed bundles

Open
#69,905 2 comments 0 reactions 0 assignees View on GitHub
area:core area:task-sdk area:ts-sdk kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Background

#69295 adds `airflow-ts-pack`, which embeds the bundle manifest as a single first-line comment in `bundle.mjs` (`//# airflowMetadata=`); the Node coordinator reads only that head line (bounded `readline()`, 1 MiB cap).

The Go SDK's AFBNDL01 executable bundle format provides two things the TS format does not (see the Trailer Layout section of `task-sdk/docs/executable-bundle-spec.rst`):

- **Integrity validation** — a SHA-256 digest over the code region, verified by the scanner before the bundle is used.
- **An embedded source region** — the original entrypoint (`.ts` / `.js`), so the UI can display the code; needed for the native TypeScript Dag feature (#69288).

Review on #69295 agreed to keep that PR to the packing flow and refine the packing spec in a follow-up. Since `bundle.mjs` must remain directly runnable by `node`, the structure cannot be a binary EOF trailer like AFBNDL01; instead the head comment can carry the layout — a format version, start/end offsets for each section, and the digest.

### What needs to happen

1. Specify a versioned head-comment layout for `.mjs` bundles recording section offsets (bundled code, embedded source, metadata) and a SHA-256 digest of the code region, documented alongside AFBNDL01 in `task-sdk/docs/executable-bundle-spec.rst`.
2. Update `airflow-ts-pack` (`ts-sdk/src/cli/pack.ts`) to emit the new layout and embed the original entrypoint source.
3. Update the Node coordinator (`task-sdk/src/airflow/sdk/coordinators/node/coordinator.py`) to verify the digest during scan — log and skip on mismatch, with the same `(path, inode, mtime, size)` result caching as the AFBNDL01 reader algorithm — and expose the embedded source.

### Acceptance criteria

- A packed `bundle.mjs` remains directly runnable with `node bundle.mjs`.
- The coordinator logs and skips a truncated or modified bundle on digest mismatch.
- The embedded entrypoint source is retrievable for display.
- `node-bundle-spec.rst` documents the TS / JS layout with its own format version.

### Context

- Originating PR: #69295
- Review threads: metadata placement (trailer moved to head): https://github.com/apache/airflow/pull/69295#discussion_r3520504501 ; deferred spec refinement ("we can refine the packing spec in the follow-up"): https://github.com/apache/airflow/pull/69295#discussion_r3551043871
- Go parallel: AFBNDL01 trailer layout, `task-sdk/docs/executable-bundle-spec.rst`
- Related: #69288 (TypeScript SDK: native TypeScript Dag declaration — the source-viewing consumer)

Contributor guide

Open the contributing guide

Research direction

Start with the Trailer Layout section in task-sdk/docs/executable-bundle-spec.rst, then inspect ts-sdk/src/cli/pack.ts and task-sdk/src/airflow/sdk/coordinators/node/coordinator.py alongside the AFBNDL01 reader algorithm. Define the versioned head-comment layout, packing and scan behavior from the acceptance criteria; done means runnable bundles, digest mismatch logging and skipping, and retrievable embedded source, with node-bundle-spec.rst updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, python, typescript
Domain
backend, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.