influxdata / influxdata/kapacitor

Influxdata catch-up load using Kapacitor

Open
#1,814 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I decided to migrate my weather station data processing from Mysql to InfluxDb.
I keep the full history of data both detail and aggregated. InfluxDb simplifies everything and works really great.

There is just one issue I am stuck with. The problem is simple and has a simple solution in any other database.

I have a Linux tablet collecting the data from all the sensors. Thanks to the batteries, it collects the data even in case of a power or network outage so there are no gaps in the measurement history.

Then there is another Linux server processing the data and serving as sort of weather station data warehouse. This one is occasionally getting disconnected from the network for many reasons. It can be for few minutes, hours or even days. And after such event, the server has to be loaded with all the missed data.

In Mysql, the automated solution is pretty simple. There is one query targetting the tablet database and loading variable Max insert date and then the insert query targetting the server which is filtered by the variable.

Using SELECT INTO OUTFILE and LOAD DATA LOCAL INFILE the performance is good even for large data collected in several days.

Perhaps I am too focused on the prior solution and cannot see the other options. But I am really stuck.

I expected to achieve something similar using Kapacitor configured to access both tablet and server InfluxDb. Well, I can load the data from one to the another using both stream and batch approach

```
stream
|from()
.database('landing')
.retentionPolicy('one_month')
.measurement('nonadditive')
.groupBy(*)
|influxDBOut()
.cluster('ubuntu')
.database('landing')
.retentionPolicy('one_month')
.measurement('nonadditive')
.precision('s')

batch
|query('SELECT * FROM "landing"."one_month"."nonadditive"')
.cluster('lubuntu')
.period(1m)
.every(1m)
.groupBy(*)
|influxDBOut()
.cluster('ubuntu')
.database('landing')
.retentionPolicy('one_month')
.measurement('nonadditive')
.precision('s')
```
Where the stream approach could cope with some short outages shorter than one minute. But That's not enough. So I tried to focus more on batch approach, introduce
the variable and use it to filter the data
```
var last_time = batch
|query('select last(value),time from "landing"."one_month"."nonadditive"')
.every(1m)
```
but I always got stuck here, trying many things to transform the variable node to the timestamp value required for the next query.
```
'SELECT * FROM "landing"."one_month"."nonadditive" WHERE time >=' + last_time
```
Is there any such possible solution which I am blind to see?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The relevant entry points are the Kapacitor stream and batch examples plus the query(), influxDBOut(), and variable expressions shown in the issue. Start by checking Kapacitor’s task-language documentation for whether a query result can feed a later query and whether cross-cluster catch-up is supported. Done means documenting a reproducible supported task or confirming the limitation and required behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, mysql, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.