influxdata / influxdata/kapacitor

Flush flatten node when values for all pipelines are received

Open
#1,933 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Seems like the flatten node is waiting for new data with greater timestamp to flush the present data.

Below is one sample simplified tick script that we use. Data arrives at a frequency of 10 seconds. So in this case, though the flatten node has all the data for time 't1', it waits for the new data 't1 + 10s' to arrive, to flush the 't1' data down the pipeline. There might be valid use cases in which it might be useful to wait in this way.

**Questions/problems that we have:**

- In our case, it is very important that we process the data as and when it arrives without any delays. Is there a way to get around this? Is there a provision to flush the data at flatten node?

- Also since the data is being held in the memory, in scale scenarios Kapacitor uses huge amount of memory. For a single device, on a limited scale scenario, it used approximately 50GB of RAM. Think of same scenario being applied for multiple devices. The memory usage multiplies by the factor of number of devices.

```
var var_in_pkts = stream
|from()
.database('Test')
.retentionPolicy('autogen')
.measurement('raw')
|eval(lambda: int("in-pkts"), lambda: string("index"), lambda: string("name"))
.as('in', 'id', 'name')
| log()
.prefix('in_stream')
var var_out_pkts = stream
|from()
.database('Test')
.retentionPolicy('autogen')
.measurement(''interfaces'')
|eval(lambda: int("out-pkts"), lambda: string("index"), lambda: string("name"))
.as('out', 'id', 'name')
| log()
.prefix('out_stream')
var joined_fields = var_in_pkts| union(var_out_pkts)
|groupBy('name', 'id')
|flatten()
.tolerance(1s)
|default()
.field('in-pkts',-2147483648)
.field('out-pkts',-2147483648)
|eval(lambda: string("id"), lambda: int("in"), lambda: string("name"), lambda: int("out"))
.as('id', 'in', 'name', 'out')
.tags('id', 'name')
|log()
.prefix('joined_fields')
joined_fields
|influxDBOut()
.measurement('processed')
.database('Test')
.flushInterval(100ms)
```

**Below are the setup details:**
OS: Ubuntu 16.04.2
RAM: 128 GB
Kapacitor version: 1.4.1 (Tested scenarios of pre-built package directly on host and on docker)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names the flatten() node and provides a simplified TICK script, but no source files or tests. Start by tracing the flatten node's buffering and timestamp handling, then reproduce the script's delayed flush and memory growth. Done requires an agreed flush condition for complete pipeline values and tests covering that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.