influxdata / influxdata/kapacitor
Feature Request: Batch writes to InfluxDB when multiple InfluxDB out nodes are used in a task
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
# The problem
Currently, the following tickscript will make 3 individual HTTP requests.
```
batch
|query('...')
|influxDBOut()
.database('mydb')
.retentionPolicy('myrp')
batch
|query('...')
|influxDBOut()
.database('mydb')
.retentionPolicy('myrp')
batch
|query('...')
|influxDBOut()
.database('mydb')
.retentionPolicy('myrp')
```
This is problematic because this can generate excess HTTP load on InfluxDB. On a node by node basis you can set the `flushInterval` and `buffer`, but for tasks that run sparsely with only a few points, data would need to be buffered for a long time. This leaves users susceptible to data loss if the instance crashed or was restarted in that timeframe.
# Outline of Possible Solution
I propose that we extend the functionality of the `influxDBOut` node to batch data with a common database and retention policy across tasks (as well as nodes in a task). Ideally users would have the option to bypass this functionality and allow nodes to make requests themselves.
Contributor guide
Research direction
Start by locating the influxDBOut node implementation and its handling of flushInterval and buffer, then trace how separate nodes create HTTP requests. Define how batching would work across nodes and tasks sharing a database and retention policy, including the proposed bypass option. Done should include an agreed design and tests for both batching and independent requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100