apache / apache/beam

Allow user provided function to extract custom timestamp from payload in pubsubIO

Open
#18,364 0 comments 0 reactions 0 assignees View on GitHub
gcp io java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently the PubsubIO runner only allows the caller to set a custom timestamp if the timestamp is defined in the attributes of the message. This can be problematic when the user does not control the publisher. In such a case, proper windowing of data requires the timestamp to be pulled out of the message payload.

Since a payload could have arbitrary data, the user would have to provide a Function() that would extract the timestamp from the payload:

PubsubIo.Read.timestampLabel(Function extractor);

Imported from Jira [BEAM-1936](https://issues.apache.org/jira/browse/BEAM-1936). Original Jira may contain additional context.
Reported by: berkoben.

Contributor guide

Open the contributing guide

Research direction

Start by locating the PubsubIO timestampLabel API and its runner implementation, then review BEAM-1936 for additional context. Define how a user-provided function extracts timestamps from message payloads and verify that the resulting timestamps support correct windowing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.