Pubsub: add DirectRunner support for id_label and timestamp_attribute in Python SDK
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
At least for publishing (and maybe pulling) messages, non-Dataflow-based sources and sinks for Pub/Sub use the [public API](https://cloud.google.com/pubsub/docs/publisher) for Pub/Sub, which doesn't support id_label and timestamp_attribute settings.
Publishing:
id_label - add an attribute to each message with a unique value
timestamp_attribute - add an attribute to each message with the publishing time as its value
Pulling:
id_label - use the value of this message attribute to deduplicate messages
timestamp_attribute - use the value of this message attribute as the element's timestamp
Implementation details: could probably create a pubsubio.py module, for reuse with other runners (i.e. implement Pub/Sub IO as PTransforms and not NativeSinks and Sources).
Imported from Jira [BEAM-4275](https://issues.apache.org/jira/browse/BEAM-4275). Original Jira may contain additional context.
Reported by: udim.
Contributor guide
Assessment
This issue has not been assessed yet.