Zero-copy / shared memory support

@PLeVasseur is already working on this.

Since Apr 10, 2025.

# Intro

We discussed in last time's Eclipse uProtocol meeting whether to use Protocol Buffers or something else for our serialization mechanism.

I'll outline some considerations here we can look at together.

# What's in a serialization mechanism?

When we talk about serialization mechanisms, we want an approach that gives us a few things at least:

1. Serialization and deserialization are cheap (memory, processing perspective)
2. Ability to represent the schema of an object in a language-agnostic way (usually there's a generator which generates language bindings)
3. Ability to extend an object schema and remain backward compatible
4. Multiple programming languages supported

However, a ~~fourth~~ fifth thing is very useful when we talk about shared-memory communication:

5. Ability to cast the binary representation directly to an object

## [Protocol Buffers](https://protobuf.dev/)

- [x] Serialization and deserialization are cheap (memory, processing perspective)
- [x] Ability to represent the schema of an object in a language-agnostic way (usually there's a generator which generates language bindings)
- [x] Ability to extend an object schema and remain backward compatible
- [x] Multiple programming languages supported
- [ ] Ability to cast the binary representation directly to an object

Protocol Buffers is a reasonable choice of serialization mechanism for inter-device communications since as soon as you have to hit the wire, the time to delivery is dominated by transfer across the wire.

## [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2)

(Look at this as a stand-in for other similar solution as well, but I like that iceoryx2 is written in Rust, personally!)

- [x] Serialization and deserialization are cheap (memory, processing perspective)
- [ ] Ability to represent the schema of an object in a language-agnostic way (usually there's a generator which generates language bindings)
- [ ] Ability to extend an object schema and remain backward compatible
- [x] Multiple programming languages supported
- [x] Ability to cast the binary representation directly to an object

iceoryx2 has cheap serialization and deserialization as essentially none is done by casting to a usable object from a shared memory buffer.

iceoryx2 also supports communication across Rust (stable) and C and C++ (beta) with more languages [planned](https://github.com/eclipse-iceoryx/iceoryx2?tab=readme-ov-file#language-bindings).

However, it appears to be lacking a way to address 2. & 3.

For 2. It may be an interesting point to work together with them on e.g. an extension to `protoc` that would generate the language bindings from .proto files (with some restrictions possibly).

For 3. this may involve generating from `protoc` in a smart way such that it's possible to still have an extensible type that's understood when casting back to an object from shared memory. May be tricky?

# A unified serialization mechanism or not?

There are two possible roads -- a unified serialization mechanism or allowing specialized ones for certain transports.

## Unified - iceoryx2

In this case, we'd formalize around using pieces of `iceoryx2` to perform all serialization and deserialization whether inter-device or intra-device.

As highlighted above, there are some pieces that appear lacking from `iceoryx2` which could be good items to bring to them for inclusion or to work on together.

In that sense, this is not a path we can immediately embark upon.

## Unified - Protocol Buffers

I don't see a way to bridge the gap for an appealing shared memory solution with Protocol Buffers, so I'd say this is out.

## Specialized

In this case we'd use:
* Protocol Buffers for inter-device communication
* iceoryx2 for intra-device communication supported by a up-transport-iceoryx2-rust

### How would this work?

Within the uStreamr we would have it configured with both:
* inter-device transports, e.g. up-transport-zenoh-rust or up-transport-mqtt5-rust
* an intra-device transport: up-transport-iceoryx2-rust

### Challenges?

We would need a means of understanding which schemas correspond to which messages within up-transport-iceoryx2-rust to work around 2. & 3. such as a configuration file. Said configuration file would need to be updated if the schema of messages being passed from an external device into our device were changed.