apache / apache/beam

Pubsub: add DirectRunner support for id_label and timestamp_attribute in Python SDK

Open
#18,939 0 comments 0 reactions 0 assignees View on GitHub
bug core direct P3 python runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

At least for publishing (and maybe pulling) messages, non-Dataflow-based sources and sinks for Pub/Sub use the [public API](https://cloud.google.com/pubsub/docs/publisher) for Pub/Sub, which doesn't support id_label and timestamp_attribute settings.

Publishing:
id_label - add an attribute to each message with a unique value
timestamp_attribute - add an attribute to each message with the publishing time as its value

Pulling:
id_label - use the value of this message attribute to deduplicate messages
timestamp_attribute - use the value of this message attribute as the element's timestamp

 

Implementation details: could probably create a pubsubio.py module, for reuse with other runners (i.e. implement Pub/Sub IO as PTransforms and not NativeSinks and Sources).

Imported from Jira [BEAM-4275](https://issues.apache.org/jira/browse/BEAM-4275). Original Jira may contain additional context.
Reported by: udim.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.