Limitations of serialization implementation regarding shared references

With the added serde derives, in PR #26 it came up on zulip that the serialization of images is "not ideal", but useful enough to keep for now but might eventually require a breaking change.

The general issue is that Image Blobs may be shared, which when serialized will duplicate the data.
This then deserializes to multiple images each having their own unique ID.

serde

The most relevant issue I found in serde is the following involving the serde_state crate
which adds on top of serde SerializeState and DeserializeState traits, alas this is an experimental fork of serde
which is not in the maintenance goldilocks zone at 110 commits ahead and 745 commits behind serde itself.

https://github.com/serde-rs/serde/issues/881

I guess my recommendation for the current state would be if serializing shared images serialize them to a map/vector
and reference those by key or index.

Alternatives to serde

Rkyv by default does the right thing for serializing shared pointers
It isn't really comparable to serde though in that it is a fixed data format rather than a data model supporting multiple data formats.

I think it might be more ideal, because it would be both zero copy, and we don't have to work around the data model to avoid duplicating shared references.

There is one issue which gives me some pause, that I would probably want to fix/work on before integrating it https://github.com/rkyv/rkyv/issues/285 because rkyv serializes enum variants by their representation value rather than nominally. Simply reordering enums or adding variants that aren't appended to the current variants can cause compatibility hazards.

No source files or tests are identified. Start by reviewing PR #26 and the linked serde issue #881, then compare the proposed serde map/vector approach with rkyv and issue #285. Done requires an agreed serialization design that preserves shared Image Blob references and addresses compatibility concerns.