influxdata / influxdata/kapacitor

Isolate Tasks that consume the same DBRP

Open
#880 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Currently all tasks that consume data from the same DBRP are coupled upstream. Where if one task slows down the rest of the tasks will have to wait.

We need a mechanism to isolate the tasks.

An initial implementation that comes to mind is to copy and buffer the points per task. But a buffer by itself won't really solve the issue since if one task can't keep up with the throughput then eventually it will back up the system again. We will have to drop points to that buffer. Using a fixed size buffer and dropping writes into the per task buffers you can isolate the tasks and only the slow tasks will loose data.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how tasks consuming the same DBRP are coupled upstream and locate the task fan-out or buffering entry point. Compare the proposed per-task fixed-size buffers and dropping behavior, then define done as slow consumers no longer blocking others while only their own buffered points are dropped; no files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.