apache / apache/pulsar-site

Primitive types schema serialization is ambiguous

Open
#1,142 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
46
Forks
217
Avg merge
1h
Merged PRs (30d)
3

Description

I have looked at a lot of pages, but I have been unable to find a definite answer as to how implementers of the clients should ensure that primitives that are serialized can be deserialized by any other client.

https://pulsar.apache.org/docs/next/schema-understand/#primitive-type

e.g.

INT64 - takes up 8 bytes in memory and would be serialized to 8 bytes if you do not use variable length serialization.

But since the endianness of the computer serializing and deseralizing the bytes controls the order of the actual bytes - then it would make sense that this project defines explicitly how numeric values must be serialized - so implementers does not have to guess.

I am aware that within the protocol of pulsar itself it is stated to use BIG endian for the payload commands size:
https://pulsar.apache.org/docs/next/developing-binary-protocol/

But imagine someone creates a producer that simply sends raw INT64 values onto a topic - then if that happens on a little endian computer, the bytes would be in a different order, and if a big endian computer deserializes the value - it will be incorrect.

Here I am specifically talking about the scenario where producer and consumer are not running the same type of client, where there are no guarantee that the two different libraries have decided to take endianness into account and decide on the same order.

This might just be me being unable to find it - but if not - then I think it would make sense if this project explicitly states how the primitive schema types must be serialized.

I am aware that most would probably be serializing values in either proto or something else where the serialization format would already be defined.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked Primitive Type schema documentation and the binary protocol documentation, then compare what each says about numeric serialization and endianness. Done means the site explicitly documents the required primitive wire format, or clearly states that the behavior is not defined and explains the interoperability implications.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
44/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.