apache / apache/beam

Revisit timestamp and duration representation

Open
#19,922 3 comments 0 reactions 0 assignees View on GitHub
beam-model improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

The current proto representation of timesetamp and durations in Beam is either raw int64s or the well-known Google protobuf types "google.protobuf.timestamp" and "google.protobuf.duration". Apache Beam uses int64 MAX and MIN as sentinel values for an ****inf watermark and -inf watermark. However, the google.protobuf.timestamp is compliant with RFC3339, meaning it can only represent date-times between 0001-01-01 and 9999-12-31. This is not the same as the int64 MAX and MIN representation. The questions remain:
* What should the timestamp and duration representations be?
* What units should the timestamps and duration be? Nanos? Micros?
* What algebra is allowed when dealing with timestamps and durations? What is needed?

See:
* [https://lists.apache.org/thread.html/c8e7d8dc7d94487fae23fa2b74ee61aec93c94abbcbef3329ffbf3bd@%3Cdev.beam.apache.org%3E](https://lists.apache.org/thread.html/c8e7d8dc7d94487fae23fa2b74ee61aec93c94abbcbef3329ffbf3bd@%3Cdev.beam.apache.org%3E)
* [https://lists.apache.org/thread.html/27fe9aa5b33dbee97db1bc924ee410048137e4fe97d9c79d131c010c@%3Cdev.beam.apache.org%3E](https://lists.apache.org/thread.html/27fe9aa5b33dbee97db1bc924ee410048137e4fe97d9c79d131c010c@%3Cdev.beam.apache.org%3E)

 

Imported from Jira [BEAM-8738](https://issues.apache.org/jira/browse/BEAM-8738). Original Jira may contain additional context.
Reported by: rohdesam.

Contributor guide

Open the contributing guide

Research direction

Begin with Apache Beam's proto timestamp and duration representations, then review the linked mailing-list discussions and BEAM-8738 context. Done means the timestamp and duration formats, units, sentinel handling, and permitted algebra have a decided specification and clear implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.