apache / apache/beam

PubsubIO support for Simple Binary Encoding (SBE)

Open
#20,957 0 comments 0 reactions 0 assignees View on GitHub
gcp io java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

We currently have a request in Dataflow templates for Pub/Sub -\> BigQuery where the Pub/Sub topic contains SBE data. However, it doesn't seem that the PubsubIO currently supports that. Ideally, we would be able to handle cases where all we have is a schema, not precompiled classes, similar to PubsubIO#readAvroGenericRecords[1].

 

[1] [https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L515](https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L515)

 

Imported from Jira [BEAM-12298](https://issues.apache.org/jira/browse/BEAM-12298). Original Jira may contain additional context.
Reported by: zhoufek.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java around the referenced readAvroGenericRecords API. Compare that entry point with the requested schema-only SBE support, then identify the relevant PubsubIO reading path and document what behavior would demonstrate completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.