influxdata / influxdata/kapacitor
Feature request: Combine latest ticks from multiple sources
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Allows for combining data with uneven timestamps.
The behaviour:
1. Across N sources, keep track of the current values on each one
2. Wait until all sources have produced a tick
3. When a tick arrives from a given source:
1. Remember the values from this tick as the current values for this source
2. Output the current values across all sources
It's different from a join with tolerance because:
- It doesn't lose precision on the output. For example, if source A ticks every millisecond and B ticks daily, you get every raw tick from A, each one annotated with today's values from B.
- The tolerance window is effectively infinite. As long as source B has ticked at some point within the stream, it will be reflected in the output.
It could be implemented as a join if join nodes supported fill(previous). The InfluxQL GROUP BY time(..) fill(previous) gives the right output.
Background: it's for financial time series. These can change on a scale of milliseconds or nanoseconds (at uneven intervals); daily (but not necessarily every day, e.g. weekends and holidays); or once (static data). The ability to combine the latest ticks across streams, regardless of timestamp, would make it possible to join such series.
Contributor guide
Research direction
Start by reviewing the existing join nodes and the fill(previous) behavior referenced in the issue. Trace how ticks from multiple sources are represented, then define the implementation boundary and tests for waiting for an initial tick from every source, retaining each source's latest values, and emitting on every subsequent tick.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100