awslabs / awslabs/flowgger

Enhance kafka commit

Open
#26 1 comment 1 reaction 0 assignees View on GitHub
help wanted question
Dominant language
Rust
Stars
879
Forks
61
PR merge metrics
No merged PRs in 30d

Description

Hello,

I think, it could be very useful to enhance "Kafka output" with a kind of timer option and associate it with the actual _kafka_coalesce_ parameter.

```
kafka_coalesce=100
kafka_flush_interval=5s
```

If one of theses conditions is met, commit the batch to Kafka.

Thanks

Contributor guide

Open the contributing guide

Research direction

Start by locating the Kafka output implementation and the handling of kafka_coalesce; trace how batches are committed and where a flush interval could be integrated. Confirm the expected behavior for kafka_flush_interval=5s, then verify that a batch is committed when either the record-count or timer condition is met.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, rust
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.