apache / apache/airflow

Cross-SDK DagSerialization compatibility CI for the Go SDK

Open
#66,942 0 comments 0 reactions 1 assignee Claimed by @jason810496 View on GitHub
area:CI area:dev-tools go-sdk kind:feature kind:task priority:low
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Background

ADR 0003 explicitly promises:

> The Go SDK gains a `serde` package that performs this encoding from
> `bundlev1.Bundle` / `bundlev1.Task`, validated against
> `validation/serialization/test_dags.yaml` (the same fixture set the
> Java SDK uses), so the Go and Java outputs are byte-identical for
> shared inputs.

The Go `serde` package exists
(`go-sdk/pkg/execution/serde.go`, `serde_test.go`) but the cross-SDK
parity hook does **not** - there is no prek/CI step that runs the
Go serializer against the shared `test_dags.yaml` and diffs the output
against Python's `SerializedDAG.serialize_dag` like the Java SDK does
via `compare.py`.

### What needs to happen

1. Add a Go-side serializer driver that consumes
`validation/serialization/test_dags.yaml` and emits a JSON file
per Dag fixture (parallel to the Java
`SerializationCompatibilityTest`).
2. Wire it into `scripts/ci/prek/` so the existing `compare.py` runs
against `serialized_go.json` alongside `serialized_java.json`.
3. Fail the check on any field-level divergence.

### Acceptance criteria

- A regression in `SerializedDAG.serialize_dag` shape that Go doesn't
follow fails CI before merge.
- The Go and Java SDKs share the fixture file (no Go-only copy).

### Context

- ADR: `go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md`.
- Java parallel: #65959 introduced
`check_java_serialization_compatibility.py`; we extend the same
pattern.
- Coordinator-distribution prek hooks tracking issue: #66515.

---
Drafted-by: Claude Code (Opus 4.7) (no human review before posting)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.