ConduitIO / ConduitIO/conduit-commons

perf: reduce allocations in structured-payload record conversion (map[string]any ↔ structpb.Struct)

Open
#269 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3
Forks
3
Avg merge
1d 24m
Merged PRs (30d)
3

Description

Committed benchmarks in `conduit-connector-protocol` (`pconnector/v2/record_marshal_bench_test.go`, added in ConduitIO/conduit-connector-protocol#260) show that **structured record payloads cost ~5–7× raw payloads** on the per-record marshal/unmarshal hot path — the hottest path in the whole system (every record crosses it, twice: source→engine and engine→destination).

## Numbers (Apple M3 Max, single-record batches, `-benchtime 2s`)

| Path | raw | structured | ratio |
|---|---|---|---|
| Marshal (record → proto → bytes) | ~1.3 µs, 29 allocs | ~7.8 µs, **125 allocs** | ~5.5× |
| Unmarshal (bytes → proto → record) | ~1.5 µs, 46 allocs | ~8.5 µs, **201 allocs** | ~5× |

## Where the cost is
The overhead is isolated to the `opencdc` record conversion in this repo — `Record.ToProto`/`FromProto` and `Change.ToProto`/`FromProto` — specifically the `map[string]any` ↔ `structpb.Struct` conversion for structured payloads (`opencdc.StructuredData`). `structpb` allocates a wrapper `*structpb.Value` per field (and recursively per nested field/element), which is why nested structured records dominate.

## Direction
- Profile `ToProto`/`FromProto` for structured payloads and reduce per-field allocations (candidates: a custom, allocation-lean `map[string]any` ↔ proto encoding instead of `structpb`; pooling; or a flatter wire representation for structured data).
- Any change here is on a **serialized/protocol-adjacent** path — must preserve exact round-trip semantics (the protocol repo's round-trip tests + the new benchmarks are the guard). No wire-format break without a versioning discussion.
- Re-run the protocol benchmarks before/after to quantify the win (the committed benchmarks make this a clean regression/benefit measurement).

Not urgent, but it's the single highest-leverage micro-optimization in the data path — it benefits every structured-record pipeline.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with opencdc Record.ToProto/FromProto and Change.ToProto/FromProto, focusing on map[string]any and structpb.Struct conversion for structured payloads. Run pconnector/v2/record_marshal_bench_test.go and the protocol round-trip tests, then compare allocations and timings while preserving exact round-trip semantics and avoiding an unversioned wire-format change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.