apache / apache/pulsar-client-go

Inefficient data encoding when using avro schema

Open
#945 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

Situation:

1. Using an avro schema definition where at least one field is of type byte.
2. Providing data, in the form of a native golang struct with a corresponding property of type []byte

It is expected that the []byte property should be encoded as-is into the binary payload using the avro codec.

#### Actual behavior

In this situation, when encoding the native golang struct into a pulsar payload using the function here: https://github.com/apache/pulsar-client-go/blob/d9b18d0690c15d5da94c9f324d9c2618e3d60bfd/pulsar/schema.go#L253, it first encodes the []byte type into a base64 string during the json.Marshal. Then after that it is converted into binary using the avro codec. This final binary data is significantly larger in size than it would have been if it was transmitted directly as the raw bytes.

Here is an example:
Original golang byte array: [172 12 53 97 9 70 89 247 94 3 56 242 127 146 9 209]
Base64 encoded text from byte array: rAw1YQlGWfdeAzjyf5IJ0Q==
The byte array of the final encoded binary payload (which is just the byte array representation of the base64 encoded string): [114 65 119 49 89 81 108 71 87 102 100 101 65 122 106 121 102 53 73 74 48 81 61 61]

In this example the encoded payload that the pulsar-client-go transmits to the pulsar queue is 50% larger in terms of bytes. This can lead to a dramatic loss to performance when throughput is the bottleneck.

#### Steps to reproduce

Encode a []byte object using the avro schema encode function here https://github.com/apache/pulsar-client-go/blob/d9b18d0690c15d5da94c9f324d9c2618e3d60bfd/pulsar/schema.go#L253 and look at the output.

#### System configuration
commit 504e589c438d35a6165f48148778b84565da8d7e

Contributor guide

Open the contributing guide

Research direction

Start in pulsar/schema.go around line 253 and reproduce the Avro encoding with a Go struct containing a []byte field. Compare the transmitted payload with the original bytes; the issue is complete when byte fields are encoded without the unnecessary base64 expansion and the resulting payload size matches the expected raw-byte representation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.