Pubsub message size limit not taking size increase from base64 encoding into account
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` bytes. This however does not take into account the size increase of base64 encoding the messages after the flush. Base64 encodes each set of three bytes into four bytes.
Therefore the 'true' size limit placed on the unencoded batch should be
x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes
Imported from Jira [BEAM-8145](https://issues.apache.org/jira/browse/BEAM-8145). Original Jira may contain additional context.
Reported by: Primevenn.
Contributor guide
Research direction
Start by locating the PubSubIO implementation and its existing batch-size handling. Verify how the batch size relates to the later base64 encoding, then update the limit so the encoded batch remains within 10 MiB. Done means the unencoded limit accounts for the stated 4-to-3 size increase and relevant PubSubIO tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100