influxdata / influxdata/kapacitor

Feature Request: Batch writes to InfluxDB when multiple InfluxDB out nodes are used in a task

Open
#1,375 0 comments 2 reactions 0 assignees View on GitHub
enhancement pm/enterprise readiness
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

# The problem
Currently, the following tickscript will make 3 individual HTTP requests.

```
batch
|query('...')
|influxDBOut()
.database('mydb')
.retentionPolicy('myrp')

batch
|query('...')
|influxDBOut()
.database('mydb')
.retentionPolicy('myrp')

batch
|query('...')
|influxDBOut()
.database('mydb')
.retentionPolicy('myrp')
```

This is problematic because this can generate excess HTTP load on InfluxDB. On a node by node basis you can set the `flushInterval` and `buffer`, but for tasks that run sparsely with only a few points, data would need to be buffered for a long time. This leaves users susceptible to data loss if the instance crashed or was restarted in that timeframe.

# Outline of Possible Solution

I propose that we extend the functionality of the `influxDBOut` node to batch data with a common database and retention policy across tasks (as well as nodes in a task). Ideally users would have the option to bypass this functionality and allow nodes to make requests themselves.

Contributor guide

Open the contributing guide

Research direction

Start by locating the influxDBOut node implementation and its handling of flushInterval and buffer, then trace how separate nodes create HTTP requests. Define how batching would work across nodes and tasks sharing a database and retention policy, including the proposed bypass option. Done should include an agreed design and tests for both batching and independent requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.