temporalio / temporalio/sdk-typescript

Add @bufbuild/protobuf (protobuf-es) payload converters

Open
#2,045 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
917
Forks
224
Avg merge
3d 16h
Merged PRs (30d)
43

Description

Is your feature request related to a problem? Please describe.

The SDK ships payload converters for protobufjs only (@temporalio/common/lib/protobufs). Users who generate their proto types with @bufbuild/protobuf v2 can't pass those types to a Temporal client or workflow without writing their own payload converter. The two libraries produce incompatible message shapes (Schema + create(Schema, init) vs Message.create(init)), so the existing converters can't be reused.

Describe the solution you'd like

A parallel converter trio under @temporalio/common/lib/protobufs-es, mirroring the existing …/protobufs exports:

  • ProtobufEsBinaryPayloadConverter
  • ProtobufEsJsonPayloadConverter
  • DefaultPayloadConverterWithProtobufsEs

Today — every protobuf-es user writes their own (~150 LOC) converter:

// payload-converter.ts
import { create, toBinary, fromBinary, toJson, fromJson, createRegistry } from '@bufbuild/protobuf';
// ...custom Binary + JSON converters, composite ordering,
//    messageType metadata, Registry forwarding for google.protobuf.Any...

Proposed — first-class support, matching the protobufjs ergonomics:

// payload-converter.ts
import { createRegistry } from '@bufbuild/protobuf';
import { DefaultPayloadConverterWithProtobufsEs } from '@temporalio/common/lib/protobufs-es';
import { MyMessageSchema } from './gen/messages_pb';

export const payloadConverter = new DefaultPayloadConverterWithProtobufsEs({
  registry: createRegistry(MyMessageSchema),
});

Sketch of the desired properties:

  • @bufbuild/protobuf as an optional peer dependency so users on the protobufjs path pay nothing.
  • Wire-compatible with the existing protobufjs converters:
    • A payload produced by either converter (binary or JSON) round-trips correctly through the other.
    • Same encoding markers (binary/protobuf, json/protobuf) and fully-qualified messageType metadata.
    • Composite ordering matches DefaultPayloadConverterWithProtobufs (Undefined → Binary → ProtobufJson → ProtobufBinary → Json), so on-the-wire converter selection is identical.
  • Constructor accepts either a pre-built Registry or a schema array (calls createRegistry internally).
Additional context

The existing docs/protobuf-libraries.md is the SDK's original (Dec '21) decision log for protobuf library choice. It defines three criteria — (A) TS types, (B) runtime-checkable message instances, (C) spec-compliant proto3 JSON — and notes that protobufjs scores A+B but not C, which is why the SDK bolts on proto3-json-serializer and patchProtobufRoot. @bufbuild/protobuf v2 satisfies A+B+C natively, and its generated output addresses every item in that doc's "Future work" section (spec-compliant to/fromJSON, fully-qualified typename, "this is a generated file" header).

Concrete use case driving this

I'm building protoc-gen-ts-temporal and protoc-gen-rust-temporal (TypeScript and Rust analogues of protoc-gen-go-temporal) so that a single .proto file generates type-safe Temporal clients, workers, and activity stubs across languages.

I'm trying to build a Node service that accepts a webhook and starts a workflow whose worker is implemented in Rust, with no intermediate Rust HTTP server that exists only to translate the webhook into a workflow start. For that to work, the TS client and the Rust worker have to agree on the wire format of the workflow args. protoc-gen-ts-temporal is built on @bufbuild/protoc-gen-es, so its generated message types are @bufbuild/protobuf v2.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the existing converters and exports under @temporalio/common/lib/protobufs, then read docs/protobuf-libraries.md for the SDK's compatibility criteria. Add the parallel protobuf-es converter exports with optional dependency and registry support, and verify binary/JSON interoperability, metadata, and composite ordering against the existing protobufjs behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, developer-experience
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.