apache / apache/pekko

Add Flow#onBackpressureDrop() operator

Open
#1,363 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
211
Avg merge
1d 6h
Merged PRs (30d)
89

Description

Motivation:

I want to do some rate limiting and just fail the single task, eg when combined with `mapAsync`.
Currently seems I can't observe the backpressure.

`onBackpressure` is only been called once downstream backpressure.

```scala
queue = Source.queue(10240)
.groupedWeightedWithin(
dynamicDictionaryConfigs.getMaxBatchTokens(), //最大的字符数,1000 个以内
dynamicDictionaryConfigs.getMaxBatchSize(), //最大的批量大小,50 个以内
costFn, //字符数计算
Duration.ofMillis(dynamicDictionaryConfigs.getMaxBatchIntervalInMillis())) //最大的聚合时间,比如 3ms
.onBackpressure(task -> task.fail(...)) //
.buffer(dynamicDictionaryConfigs.getOvsQpsLimit() * 2, OverflowStrategy.backpressure())
.toMat(Sink.foreach(this::batchTranslate), Keep.left())
....
.run(actorSystem);
```

What do you think?

In reactor-core, there is:
- `reactor.core.publisher.Flux#onBackpressureDrop(java.util.function.Consumer)`

This is what I would like to make use.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing Flow#onBackpressure operator and comparing its behavior with reactor-core's Flux#onBackpressureDrop(java.util.function.Consumer). Trace how groupedWeightedWithin, mapAsync, and buffer propagate backpressure, then define the dropped-element callback and verify the intended behavior with focused stream tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.