influxdata / influxdata/kapacitor

Feature request: random stream access

Open
#2,356 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hello team,

We are getting to know the TICK stack experimenting with InfluxDB and Kapacitor. We are really impressed by how rich and easy the TICKscript DSL is, and also by the huge number of supported integrations.

Specifically for a use case we have, we aim at syncing an on-prem InfluxDB site with a remote Kafka-compatible one. The Kafka integration works like a charm, very easy to setup with the Kapacitor configuration file.

What we are after is basically: an InfluxDB -> Kafka mirror. Ideally we would like to stream InfluxDB from the very beginning of a set, or some time t in general. But we see in the API that `stream()` does not accept arguments, i.e. `stream(t)` or something like this.

Two questions:

- we know already that this use case might not fit the stack 100% as Kapacitor is meant for efficient event aggregation and alerting, not as a proxy to InfluxDB; if possible, could you point us to the best way to achieve the above, by just relying on TICK so no manual integrations?

- does the feature request suggested above make sense for the mission of Kapacitor?

Thank you in advance,

cc @demadred

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `stream()` API and the Kafka integration configured through Kapacitor's configuration file, then review how TICKscript represents stream starts. Done means there is a decided approach for beginning an InfluxDB stream at its beginning or at time t, or a documented conclusion that this is outside Kapacitor's scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.