microsoft / microsoft/typespec

Use JSON instead of YAML for typespec-python code model serialization

Open
#11,419 1 comment 1 reaction 1 assignee Claimed by @msyyc View on GitHub
emitter:client:python feature
Dominant language
Java
Stars
5.9k
Forks
394
Avg merge
1d 23h
Merged PRs (30d)
104

Description

## Problem

Today `@typespec/http-client-python` exchanges the code model between the TypeScript emitter and the Python generator as **YAML**. This is very slow and dominates end-to-end generation time, especially under Pyodide where PyYAML runs as pure Python.

Measured on the Azure `network` spec (large real-world spec):

| Stage | YAML (js-yaml / PyYAML) |
| --- | --- |
| JS serialize | ~1000 ms |
| Python load | ~11000 ms |
| Python dump | ~5500 ms |
| File size | 14.4 MB |

Total serialization overhead: **~29 s**.

Switching to a faster JS YAML library (`yaml`/eemeli) does not help — it is ~1.8x slower to serialize and ~70x slower to parse than `js-yaml`, and it rejects the model by default due to the alias-count guard. The bottleneck is the Python side, which only a non-YAML format fixes.

## Proposal

Replace YAML with a **reference-preserving JSON** format, using the same `$id`/`$ref` convention as the C# emitter (System.Text.Json `ReferenceHandler.Preserve`):

- First time an object is seen: `{ "$id": "N", ...properties }`
- First time an array is seen: `{ "$id": "N", "$values": [...] }`
- Any later reference to an already-seen node: `{ "$ref": "N" }`

The code model is a cyclic/shared object graph (on the network spec: 35,175 objects and 32,075 arrays, of which 3,739 objects and 2,855 arrays are shared, plus 1,022 cycle back-edges). Because `$id` must be registered *before* recursing into a node's children, cycles are encoded as a `$ref` back to the enclosing node, so cycles are preserved (the C# writer drops them, which the Python model cannot tolerate). The Python decoder then reconstructs a graph with identical shared identity and cycles, so the generator's model layer needs **zero changes**.

Expected result: **~29 s -> ~0.2 s** and file size 14.4 MB -> 8.8 MB.

## Work items

- [ ] Add `emitter/src/code-model-serializer.ts` — cycle-safe `$id`/`$ref` serializer
- [ ] Add `generator/pygen/_codemodel_json.py` — matching Python `loads`/`dumps` decoder/encoder
- [ ] Drop `js-yaml`/`@types/js-yaml` (emitter) and `PyYAML`/`types-PyYAML` (generator) dependencies
- [ ] Rename the `emit-yaml-only` emitter option to `emit-codemodel-only`
- [ ] Update CI regenerate scripts, batch runner, mypy config, and `ARCHITECTURE.md`

## Validation criteria

- Full package build (emitter + pygen wheel)
- End-to-end: compile the network spec -> `$id`/`$ref` code model -> generator produces correct output
- Python codec reconstructs an identical graph (structural metrics match) and round-trips stably
- Generator unit tests, lint, format, and mypy pass

## Reference

POC: https://github.com/microsoft/typespec/pull/11178

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.