apache / apache/pulsar-client-go
Inefficient data encoding when using avro schema
- Dominant language
- Go
- Stars
- 745
- Forks
- 389
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 3
Description
#### Expected behavior
Situation:
1. Using an avro schema definition where at least one field is of type byte.
2. Providing data, in the form of a native golang struct with a corresponding property of type []byte
It is expected that the []byte property should be encoded as-is into the binary payload using the avro codec.
#### Actual behavior
In this situation, when encoding the native golang struct into a pulsar payload using the function here: https://github.com/apache/pulsar-client-go/blob/d9b18d0690c15d5da94c9f324d9c2618e3d60bfd/pulsar/schema.go#L253, it first encodes the []byte type into a base64 string during the json.Marshal. Then after that it is converted into binary using the avro codec. This final binary data is significantly larger in size than it would have been if it was transmitted directly as the raw bytes.
Here is an example:
Original golang byte array: [172 12 53 97 9 70 89 247 94 3 56 242 127 146 9 209]
Base64 encoded text from byte array: rAw1YQlGWfdeAzjyf5IJ0Q==
The byte array of the final encoded binary payload (which is just the byte array representation of the base64 encoded string): [114 65 119 49 89 81 108 71 87 102 100 101 65 122 106 121 102 53 73 74 48 81 61 61]
In this example the encoded payload that the pulsar-client-go transmits to the pulsar queue is 50% larger in terms of bytes. This can lead to a dramatic loss to performance when throughput is the bottleneck.
#### Steps to reproduce
Encode a []byte object using the avro schema encode function here https://github.com/apache/pulsar-client-go/blob/d9b18d0690c15d5da94c9f324d9c2618e3d60bfd/pulsar/schema.go#L253 and look at the output.
#### System configuration
commit 504e589c438d35a6165f48148778b84565da8d7e
Contributor guide
Research direction
Start in pulsar/schema.go around line 253 and reproduce the Avro encoding with a Go struct containing a []byte field. Compare the transmitted payload with the original bytes; the issue is complete when byte fields are encoded without the unnecessary base64 expansion and the resulting payload size matches the expected raw-byte representation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100