Protocol pipelining via client-allocated object refs (POC: ~17-20% faster synth)
- Dominant language
- TypeScript
- Stars
- 2.9k
- Forks
- 267
- Avg merge
- 1d 25m
- Merged PRs (30d)
- 14
Description
## Context
Large CDK (Python) synths spend most of their time in the synchronous host↔kernel protocol: a strict request/response ping-pong where the host blocks on every `create`/`invoke`/`get`. Profiling (see #5166) showed each side idle ~half the time waiting for the other, while JSON serialization is only ~2% — so the lever is **overlapping round-trips**, not the wire format.
This issue tracks a working proof-of-concept of **protocol pipelining via client-allocated object references**, and the work needed to productionize it. Draft PR: (link below).
> Prototyped by **Kiro** (an AI agent), working with @mrgrain.
## Mechanism
- Any call whose declared return is a **reference type** (class/interface) is fired **without waiting**: the host mints the object id, sends it with the request, and uses a synthetic handle immediately.
- The kernel honors a client-supplied `objid` on `create`, and on `invoke`/`sinvoke`/`get`/`sget` **aliases** that id to whatever object the call actually produced — fresh or pre-existing. Multiple ids → one object is fine; kernel-side identity stays correct.
- The host drains acks lazily and only blocks at **true sync points**: value returns (e.g. reading a token string), callbacks, end of synth. A pending cap bounds outstanding requests to avoid pipe-buffer deadlock.
- ref-vs-value is decided from the generated binding's declared return type.
## Results (POC)
~2,200-resource synth, best-of-5, same runtime, **byte-identical CloudFormation output**:
| variant | wall |
|---|---|
| baseline | ~5.8–6.1 s |
| pipelined (create+invoke+sinvoke+sget) | ~4.7–4.9 s |
**~17–20% faster.** Findings:
- **create-only pipelining ≈ 0%** — interleaved synchronous invokes are barriers that drain the pipeline. Pipelining invokes too is what unlocks the win.
- **get/sget added little here** — instance `get`s in this app are all *value* reads (`bucket_name`, `queue_url`, `table_name` → token strings), which are irreducible sync points; only static `sget`s returned references.
- The remaining gap to the theoretical ~2× ceiling is the **kernel's serial work** (construct JS + dispatch), not the protocol. Pipelining overlaps the host with the kernel; once overlapped, you're kernel-work-bound.
## Remaining TODOs / gaps
Productionizing this is a real cross-cutting project:
- [ ] **Codegen instead of runtime introspection.** The POC reads the caller's return annotation at runtime (frame walk + `get_type_hints`, cached). Production should have **pacmak emit the return fqn** into the binding call (`jsii.invoke(self, "m", args, return_fqn=...)`) — faster, deterministic, and removes the frame-walk hack. Needed per target language.
- [ ] **Deferred error semantics.** A failed pipelined call surfaces at the next sync point, not the call site. Requests need tagging so errors map back to the originating call with a usable stack. The POC ignores errors on the drained path.
- [ ] **Callbacks.** Overrides (host ← kernel callbacks) are hard sync points; the pipeline must drain/coordinate around them. The POC asserts/raises if a callback appears mid-pipeline (the test app has none).
- [ ] **Object identity.** Client-minted ids mean two reads of the same underlying object yield two distinct host proxies (`a.prop is a.prop` can become False). Kernel-side identity is preserved; host-side needs a story (e.g. cache ref→proxy, or reconcile at sync points). Did not affect output here, but identity-sensitive code could differ.
- [ ] **Id-space / collision discipline.** The kernel must accept and trust client ids in a defined, non-colliding range (POC uses `fqn@<1e9+seq>`); formalize and validate.
- [ ] **Capability negotiation.** Gate on a handshake flag so old/new hosts and kernels interoperate; never send `objid` to a kernel that doesn't understand it.
- [ ] **Pending-cap tuning / backpressure.** POC uses a fixed cap of 64; needs principled backpressure to avoid pipe-buffer deadlock across platforms.
- [ ] **All language runtimes.** POC is Python-only; the kernel side is language-agnostic, but Java/.NET/Go hosts need the same client work.
- [ ] **Async (`begin`/`end`) and `del`** interactions with the pipeline are unexplored (absent in the test app).
- [ ] **Tests** for the kernel alias behavior and the host pipeline/drain/correctness; broader app coverage beyond the synthetic stack.
## Related
- #5166 — wire-protocol profiling and analysis (why JSON isn't the bottleneck; the ~2× ceiling).
Contributor guide
Research direction
Start by reading the existing Python POC and the pacmak binding-generation path around the jsii.invoke call, then inspect the kernel create/invoke/get alias handling. Run the existing synth benchmark and identify tests covering pipeline draining and correctness. Done requires agreed error, callback, identity, id-space, negotiation, backpressure, async, and multi-language behavior with kernel and host tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, go, java, python, typescript
- Domain
- backend-api-design, distributed-systems, performance, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100