compressed image transport: end-to-end benchmark report (raw Image vs CompressedCodec)

This is a prereq to @paul-nechifor webtransport/webrtc transport and frankly crazy we've waited this long to do this given how horrible sending raw images is. See benchmarks below and a few transport design decisions need to choose between

Option 1: CompressedCodec as a wrapper on PubSub that compresses inner transport on the wire and handles decoding on the other side with a param decode=True if the module on the other end accepts In[Image

initial implementation: https://github.com/dimensionalOS/dimos/pull/2814

Blueprint example:

unitree_go2_compressed_image = unitree_go2_basic.transports(
    {
        ("color_image", Image): CompressedCodec(
            LCMTransport("/color_image", CompressedImage), quality=80
        ),
    }
)

pretty ugly solution

Option 2: In Transport config layer, add params for compressed images and quality and pass in via cli with -o transports.*

# global_config.py 
  class GlobalConfig(...):
      compress_images: bool | None = None   # None = per-backend default
      image_quality: int = 75
      image_max_width: int | None = None

  # transport_factory.py
  _COMPRESSIBLE_TYPES = ("sensor_msgs.Image",)   # depth types excluded: jpeg can't carry them

  def _image_compression(msg_type: type | None, g: GlobalConfig) -> tuple[int, int | None] | None:
      """(quality, max_width) if this stream should default to compressed, else None."""
      if getattr(msg_type, "msg_name", None) not in _COMPRESSIBLE_TYPES:
          return None
      on = g.compress_images
      if on is None:
          # per-backend default: network wires compress, on-host stays raw
          on = g.transport in ("lcm", "zenoh", "webrtc")
      return (g.image_quality, g.image_max_width) if on else None

usage example:
compression enabled by default

dimos -o g.image_quality=50 run unitree-go2 # tune globally
DIMOS_COMPRESS_IMAGES=0 dimos run ...       # opt out globally

pretty clean, but less customizability / hidden for devs could be super hard to debug issues for new devs

Option 3: Some plum conversion thing that hides a decode inside all Image transports if it recieves CompressedImage instead

like:

@dispatch
  def _convert(msg: CompressedImage, ...) -> Image:
      return msg.decode()

also hidden and prob bad

Option 4: type-level, Annotated streams with some "compressionHint" - Compressed[Image] = [Image, compresssionHint(...)]

Requres more transport / blueprint arch changes, to autoconnect() etc. Otherwise probably cleanest

  class GO2Connection(Module):
      color_image: Out[Compressed[Image]]        # "I emit images, wire may compress"

  class Detector(Module):
      color_image: In[Image]                     # wants pixels - autoconnect inserts decode

  class WebRelay(Module):
      color_image: In[Compressed[Image]]         # wants bytes

end-to-end replay benchmark: raw Image vs Compressed Image

blueprints run for real in --replay mode (go2_short dataset, ~14Hz 720p camera), 60s per run, 2 reps per cell, headless. the ONLY difference between raw and codec is one blueprint pin:

.transports({("color_image", Image): CompressedCodec(LCMTransport("/color_image", CompressedImage), quality=75)})

harness: dimos/protocol/pubsub/benchmark/tool_replay_bench.py — adds BenchSink consumer modules (record every frame arrival; optionally burn N ms of real cv2 work per frame like a busy detector) + a host sampler (process-tree cpu/rss + loopback Mbit/s at 1Hz).

matrix:

• unitree-go2 — full nav stack (slam, costmap, planner), 11 workers. the heavy real blueprint.
• heavy — unitree-go2 + 4 BenchSinks each burning 20ms/frame (simulates 4 busy detector-class consumers)
• unitree-go2-basic — 4 workers, control
• profiles: clean (32 cores) and jetson (taskset -c 0-3, simulates robot-class onboard compute). wifi (tc netem) pending — needs sudo.

summary (medians, 2 reps pooled)

blueprint             profile  mode    frames    fps  lag@end   cpu%  rss MB  lo Mbps
heavy                 clean    raw        836   14.0     -4ms     24    7127      359
heavy                 clean    codec      855   14.3    -53ms      6    5733       47

heavy                 jetson   raw        674   11.4     -8ms     13    7433      352
heavy                 jetson   codec      855   14.3    -54ms      3    6726       44

unitree-go2           clean    raw        830   13.9    -56ms     17    6959      357
unitree-go2           clean    codec      855   14.3    -56ms      4    5610       48

unitree-go2           jetson   raw        376    6.5     -6ms      7    6926      351
unitree-go2           jetson   codec      854   14.3    -62ms      2    6502       43

unitree-go2-basic     clean    raw        197    3.3    -28ms      8    2765      332
unitree-go2-basic     clean    codec      854   14.3    -56ms      3    2571       31

delivered camera fps (replay publishes ~14Hz)

heavy               clean   raw   ██████████████████████████████ 14.0
heavy               clean   codec ██████████████████████████████ 14.3

heavy               jetson  raw   ████████████████████████ 11.4
heavy               jetson  codec ██████████████████████████████ 14.3

unitree-go2         clean   raw   ██████████████████████████████ 13.9
unitree-go2         clean   codec ██████████████████████████████ 14.3

unitree-go2         jetson  raw   ██████████████ 6.5
unitree-go2         jetson  codec ██████████████████████████████ 14.3

unitree-go2-basic   clean   raw   ███████ 3.3
unitree-go2-basic   clean   codec ██████████████████████████████ 14.3

charts

headline — full nav stack on robot-class compute (taskset 4 cores). green = codec, red = raw, dashed = published 14Hz:

same blueprint, clean 32-core box (delivery ties — the win is cpu/wire):

with 4 busy detector-class consumers (20ms real work per frame each), 4 cores:

the 4-worker control — raw collapses even on the clean box (receiver contention):

cpu over time, full nav stack clean profile (raw red vs codec green):

tldr;

1. on robot-class compute (jetson profile), raw loses half the camera. full nav stack pinned to 4 cores: raw delivers 6.5 of 14 fps; codec delivers all of it, steady for the whole run. with 4 busy consumers on top (heavy/jetson) raw manages 11.4 — and one raw run wedged so hard the coordinator couldnt even shut down inside 240s.

2. LCM sheds frames instead of lagging. lag-behind-playback stays flat ~10ms in ALL cells — the raw path never delivers stale frames, it just doesnt deliver them. so "latency building over time" doesnt happen on lcm; frame LOSS is the failure mode. (on a reliable transport like zenoh it would lag instead.)

3. codec uses LESS total cpu, not more. counterintuitive but consistent: raw 17-24% vs codec 4-6% (clean). shoveling 2.76MB messages through lcm (fragmentation, copies, drops) costs more cpu than jpeg encode+decode of the same frames. rss is ~1.3GB lower too.

4. wire: ~350 → ~45 Mbit/s (8x). not 20x because only color_image is pinned — lidar/global_map/costmap stay raw lcm in both modes (deliberate: they're the control, and they behaved identically).

5. unitree-go2-basic (4 workers) raw delivered only 3.3fps on a CLEAN 32-core box — worse than the full nav stack. with few workers the sink shares a contended process and 2.76MB deliveries drop at the receiving end (median inter-frame gap 141-274ms, gaps up to 2s, both reps). codec on the identical config: 14.3fps. raw lcm image delivery is fragile to receiver contention; 140KB messages are not.

repro

python -m dimos.protocol.pubsub.benchmark.tool_replay_bench \
  --blueprint unitree-go2 --mode codec --sinks 4 --work-ms 20 --duration 60 --out /tmp/bench/x
# jetson profile: prefix with `taskset -c 0-3`

Start with dimos/protocol/pubsub/benchmark/tool_replay_bench.py and the initial CompressedCodec implementation in PR 2814; run the replay benchmark using the command in the issue to verify the reported results. Done requires choosing and documenting a transport-compression approach, with its scope and acceptance criteria made explicit.