apache / apache/beam

Pubsub message size limit not taking size increase from base64 encoding into account

Open
#19,744 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` bytes. This however does not take into account the size increase of base64 encoding the messages after the flush. Base64 encodes each set of three bytes into four bytes.

Therefore the 'true' size limit placed on the unencoded batch should be
x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes

Imported from Jira [BEAM-8145](https://issues.apache.org/jira/browse/BEAM-8145). Original Jira may contain additional context.
Reported by: Primevenn.

Contributor guide

Open the contributing guide

Research direction

Start by locating the PubSubIO implementation and its existing batch-size handling. Verify how the batch size relates to the later base64 encoding, then update the limit so the encoded batch remains within 10 MiB. Done means the unencoded limit accounts for the stated 4-to-3 size increase and relevant PubSubIO tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.