consumer lag issue for slow consumers
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
I have a kafka producer which produces messages continuously but my consumer consumes every 2 seconds. Since my my consumer is slow consumer getting ERR notifications. find below configurations and API response.
**burrow.toml:**
[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="mysite.example.com"
[logging]
filename="C://Go/src/Burrow/config/logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true
[zookeeper]
servers=[ "localhost:2181" ]
timeout=6
root-path="/burrow"
[client-profile.test]
client-id="burrow-test"
kafka-version="1.1.1"
[cluster.df-test-app]
class-name="kafka"
servers=[ "localhost:9092" ]
client-profile="test"
topic-refresh=120
offset-refresh=30
[consumer.df-test-app]
class-name="kafka"
cluster="df-test-app"
servers=[ "localhost:9092" ]
client-profile="test"
group-denylist="^(console-consumer-|python-kafka-consumer-|quick-).*$"
group-allowlist=""
[cluster.df-test-clog]
class-name="kafka"
servers=[ "localhost:9092" ]
client-profile="test"
topic-refresh=120
offset-refresh=30
[consumer.df-test-clog]
class-name="kafka"
cluster="df-test-clog"
servers=[ "localhost:9092" ]
client-profile="test"
group-denylist="^(console-consumer-|python-kafka-consumer-|quick-).*$"
group-allowlist=""
[httpserver.default]
address=":8000"
[storage.default]
class-name="inmemory"
workers=20
intervals=5
expire-group=604800
min-distance=1
[notifier.default]
class-name="email"
interval=180
template-open="C://Go/src/Burrow/config/default-email.tmpl"
template-close="C://Go/src/Burrow/config/default-email-statusok.tmpl"
send-close=false
server="smtp.gmail.com"
port="587"
auth-type="plain"
username="test@gmail.com"
password="test123"
from="test@gmail.com"
to="test@gmail.com"
threshold=1
**burrow consumer group status:**
{
"error": false,
"message": "consumer status returned",
"status": {
"cluster": "df-test-app",
"group": "kvsrr.burrow.grp.3",
"status": "ERR",
"complete": 1,
"partitions": [
{
"topic": "kvsrr.burrow.topic",
"partition": 0,
"owner": "",
"client_id": "",
"status": "STOP",
"start": {
"offset": 14405033,
"timestamp": 1600009858736,
"observedAt": 1600046210000,
"lag": 147079641
},
"end": {
"offset": 14600596,
"timestamp": 1600009879054,
"observedAt": 1600046210000,
"lag": 146884078
},
"current_lag": 146884078,
"complete": 1
}
],
"partition_count": 1,
"maxlag": {
"topic": "kvsrr.burrow.topic",
"partition": 0,
"owner": "",
"client_id": "",
"status": "STOP",
"start": {
"offset": 14405033,
"timestamp": 1600009858736,
"observedAt": 1600046210000,
"lag": 147079641
},
"end": {
"offset": 14600596,
"timestamp": 1600009879054,
"observedAt": 1600046210000,
"lag": 146884078
},
"current_lag": 146884078,
"complete": 1
},
"totallag": 146884078
},
"request": {
"url": "/v3/kafka/df-test-app/consumer/kvsrr.burrow.grp.3/status",
"host": "IN01N03273"
}
}
Contributor guide
No contributing guide indexed for this repository
Research direction
The report provides burrow.toml and a Burrow consumer-status API response, but names no source file, test, or expected behavior. Start by reproducing the configuration and examining the /v3/kafka/df-test-app/consumer/.../status response for the ERR and STOP states. Done is not defined; the expected handling of a slow consumer must be established before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability-sre, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100