TimelyDataflow / TimelyDataflow/differential-dataflow
Question: how to change data timestamp for late arriving data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Reason to ask:
Due to IoT(Internet of Things) scenario, lot of chance the data will arrive with unknown delay. We need put data to right timestamp which is data collected from device rather than the time data arrives.
Question:
Usually the InputSession.insert will use the same timestamp which been advanced by advacne_to function. As restriction of the timely dataflow, we not allow advance to previous timestamp. I'd like to know how to handle such case in differential dataflow.
Thanks lot.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the InputSession.insert and advance_to behavior in the differential-dataflow and timely-dataflow documentation or source. Determine how late-arriving device timestamps are expected to be represented under timely dataflow's timestamp restrictions, then document the supported approach and its limitations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, stream-processing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100