Primitive types schema serialization is ambiguous
- Dominant language
- HTML
- Stars
- 46
- Forks
- 217
- Avg merge
- 1h
- Merged PRs (30d)
- 3
Description
I have looked at a lot of pages, but I have been unable to find a definite answer as to how implementers of the clients should ensure that primitives that are serialized can be deserialized by any other client.
https://pulsar.apache.org/docs/next/schema-understand/#primitive-type
e.g.
INT64 - takes up 8 bytes in memory and would be serialized to 8 bytes if you do not use variable length serialization.
But since the endianness of the computer serializing and deseralizing the bytes controls the order of the actual bytes - then it would make sense that this project defines explicitly how numeric values must be serialized - so implementers does not have to guess.
I am aware that within the protocol of pulsar itself it is stated to use BIG endian for the payload commands size:
https://pulsar.apache.org/docs/next/developing-binary-protocol/
But imagine someone creates a producer that simply sends raw INT64 values onto a topic - then if that happens on a little endian computer, the bytes would be in a different order, and if a big endian computer deserializes the value - it will be incorrect.
Here I am specifically talking about the scenario where producer and consumer are not running the same type of client, where there are no guarantee that the two different libraries have decided to take endianness into account and decide on the same order.
This might just be me being unable to find it - but if not - then I think it would make sense if this project explicitly states how the primitive schema types must be serialized.
I am aware that most would probably be serializing values in either proto or something else where the serialization format would already be defined.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked Primitive Type schema documentation and the binary protocol documentation, then compare what each says about numeric serialization and endianness. Done means the site explicitly documents the required primitive wire format, or clearly states that the behavior is not defined and explains the interoperability implications.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 44/100