Serialization/Deserialization API for AST

Search Terms

serialize deserialize ast node

Ref #26871
Ref #28365

Suggestion

Implement/expose an API that can serialize/deserialize AST nodes, in particular a way to serialize a source file so it can be easily transferred or stored, potentially manipulated, and hydrated back.

Somewhat related is #28365 which would allow ingestion of AST directly for certain use cases. It was suggested before in #26871 for performance reasons, but not for externalising the AST in a form that can be persisted easily.

Use Cases

Specifically in Deno, we are interested in doing some AST generation or manipulation external to the JavaScript runtime. Doing some computationally intense functions not in JavaScript can have significant improvements in speed. For example, we originally did our source map remappings for errors in JavaScript using the source-map package from Mozilla. In version 0.7+ though, Mozilla had written the mappings part of it in Rust transpiled to WASM. We were able to use that directly in the privileged side of Deno, in its native Rust and saw something like a 100x increase in performance.

At this point we don't specifically know how we would use it, though there can potentially be a need to procedurally generate some aspects like our runtime type library or do some AST transformations for bundling. We have experimented with doing this manipulation in a JavaScript runtime and wonder if we can get performance improvements by doing some stuff in Rust.

Right now, a full parse, transform and emit is the only way to "persist" code. If you want to make further changes, it is a reparse. It seems logical that this transitory state could be utilised for a lot of different advanced use cases. It also would in a way decouple the parser and the transformer externally.

There are a good amount of advanced use cases that people build on top of AST generators like acorn and estree, so it is logical that other advanced use cases could be built if serialisation/deserialisation is available.

Two big challenges are likely that there is a lot of circular references, which makes JSON a difficult serialisation format. There is a lot of hidden state in a SourceFile that isn't "owned" by the surface object. That would have to be exposed as part of a serialisation which isn't present as properties in the existing nodes.

I started to look at tsbuildinfo to see if some sort of compiler or AST state is persisted somewhere that could be used to rehydrate the state of the compiler, but really haven't looked to see if internally TypeScript can serialise a program state.

Examples

const sourceFile = ts.createSourceFile("mymodule.ts", "console.log(\"hello\");", ts.ScriptTarget.ESNext, true);

// Returns a POJO of ts.SerializedNode?
const serializedSourceFile = ts.serialize(sourceFile);

// Because it is a POJO, it can be easily serialized
console.log(JSON.stringify(serializedSourceFile));

// Accepts a ts.SerializedNode?
const sourceFile2 = ts.deserialize(serializedSourceFile);

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.

まず、既存の tsbuildinfo の処理と issue 内の createSourceFile の例を確認し、コンパイラと SourceFile のどの状態を API 境界の向こう側に渡す必要があるかを判断します。循環参照と非公開の SourceFile の状態がシリアライズ可能な表現にどのような影響を与えるかを調査します。完了には、AST または source file を外部ストレージに保存して操作したうえで round-trip できる、合意済みの公開 serialize/deserialize 設計が必要です。