confluentinc / confluentinc/kafka-tutorials

Avoid to mixup the terms "late" and "out-of-order"

Open
#297 0 comments 1 reaction 0 assignees View on GitHub
ksqlDB kstreams
Dominant language
Java
Stars
39
Forks
91
PR merge metrics
No merged PRs in 30d

Description

In Kafka Streams and kslqDB (and also other stream processing systems), the terms "out-of-order" and "late" have a very specific meaning and they are **_not_** synonyms. However, in https://kafka-tutorials.confluent.io/window-final-result/kstreams.html both are used as synonyms (there might be other tutorial with the same issue and it might be worth to double check).

The term "out-of-order data" describes events that violate event-time order, ie, their timestamp is smaller than the current "stream time". The "stream time" is the maximum seen timestamp, ie, stream-time only advances but never goes backwards. If an event is out-of-order, it is **_not_** necessarily "late" though.

The term "late data" describes events that are out-of-order **_and_** are not processed by an operator (but dropped), because the corresponding window is already closed, ie, the data is "late for the computation". Thus, late data by definition is _never_ processed and the phrase "processing late data" does not make sense. To be more specific, events are late if they arrive after the grace period passed, and thus only windowed operator might drop data as "late".

Beside cleanup up our tutorials to use the right terminology, we might even want to add some definition for both terms to make it clear what they mean.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.