ConduitIO / ConduitIO/conduit-commons
perf: reduce allocations in structured-payload record conversion (map[string]any ↔ structpb.Struct)
- Dominant language
- Go
- Stars
- 3
- Forks
- 3
- Avg merge
- 1d 24m
- Merged PRs (30d)
- 3
Description
Committed benchmarks in `conduit-connector-protocol` (`pconnector/v2/record_marshal_bench_test.go`, added in ConduitIO/conduit-connector-protocol#260) show that **structured record payloads cost ~5–7× raw payloads** on the per-record marshal/unmarshal hot path — the hottest path in the whole system (every record crosses it, twice: source→engine and engine→destination).
## Numbers (Apple M3 Max, single-record batches, `-benchtime 2s`)
| Path | raw | structured | ratio |
|---|---|---|---|
| Marshal (record → proto → bytes) | ~1.3 µs, 29 allocs | ~7.8 µs, **125 allocs** | ~5.5× |
| Unmarshal (bytes → proto → record) | ~1.5 µs, 46 allocs | ~8.5 µs, **201 allocs** | ~5× |
## Where the cost is
The overhead is isolated to the `opencdc` record conversion in this repo — `Record.ToProto`/`FromProto` and `Change.ToProto`/`FromProto` — specifically the `map[string]any` ↔ `structpb.Struct` conversion for structured payloads (`opencdc.StructuredData`). `structpb` allocates a wrapper `*structpb.Value` per field (and recursively per nested field/element), which is why nested structured records dominate.
## Direction
- Profile `ToProto`/`FromProto` for structured payloads and reduce per-field allocations (candidates: a custom, allocation-lean `map[string]any` ↔ proto encoding instead of `structpb`; pooling; or a flatter wire representation for structured data).
- Any change here is on a **serialized/protocol-adjacent** path — must preserve exact round-trip semantics (the protocol repo's round-trip tests + the new benchmarks are the guard). No wire-format break without a versioning discussion.
- Re-run the protocol benchmarks before/after to quantify the win (the committed benchmarks make this a clean regression/benefit measurement).
Not urgent, but it's the single highest-leverage micro-optimization in the data path — it benefits every structured-record pipeline.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with opencdc Record.ToProto/FromProto and Change.ToProto/FromProto, focusing on map[string]any and structpb.Struct conversion for structured payloads. Run pconnector/v2/record_marshal_bench_test.go and the protocol round-trip tests, then compare allocations and timings while preserving exact round-trip semantics and avoiding an unversioned wire-format change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100