influxdata / influxdata/kapacitor
Isolate Tasks that consume the same DBRP
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Currently all tasks that consume data from the same DBRP are coupled upstream. Where if one task slows down the rest of the tasks will have to wait.
We need a mechanism to isolate the tasks.
An initial implementation that comes to mind is to copy and buffer the points per task. But a buffer by itself won't really solve the issue since if one task can't keep up with the throughput then eventually it will back up the system again. We will have to drop points to that buffer. Using a fixed size buffer and dropping writes into the per task buffers you can isolate the tasks and only the slow tasks will loose data.
Contributor guide
Research direction
Start by tracing how tasks consuming the same DBRP are coupled upstream and locate the task fan-out or buffering entry point. Compare the proposed per-task fixed-size buffers and dropping behavior, then define done as slow consumers no longer blocking others while only their own buffered points are dropped; no files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100