Blizzard / Blizzard/node-rdkafka

Release with librdkafka >= 2.15.0 — KIP-848 (group.protocol=consumer) consumer fixes needed

Open
#1,146 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

## Summary

The latest node-rdkafka release (v3.6.1) still bundles **librdkafka 2.12.0**. librdkafka **2.15.0** (released 2026-06-30) contains important fixes for the KIP-848 next-generation consumer group protocol (`group.protocol=consumer`), which is GA since 2.12.0. We are running `group.protocol=consumer` in production through node-rdkafka and have hit consumer-reliability problems that match the bugs fixed in 2.15.0. Since node-rdkafka pins the bundled librdkafka, there is currently no npm-installable way to get these fixes.

**Request: publish a release bundling librdkafka 2.15.0 (or newer).**

## Environment

- node-rdkafka 3.6.0 / 3.6.1 (librdkafka 2.12.0)
- Brokers: Apache Kafka 4.0.0 (KRaft, Strimzi)
- Consumer config: `group.protocol=consumer`, `enable.auto.commit=false` + `commitSync()`, `auto.offset.reset=largest`
- Multiple consumer pods per group, autoscaled (frequent rebalances)

## What we observed with 2.12.0 + KIP-848

1. **Silent consumption stalls**: consumers stop fetching entirely for 1–3 minutes during group reconciliations, with no `event.error` emitted and nothing in logs, then resume.
2. **Frequent commit failures** with `Broker: The member epoch is stale` (errno 113) whenever members join/leave (autoscaling) — hundreds per day across groups.
3. **In rare cases, silently skipped offset ranges**: after reconciliation storms (e.g. a staggered `subscribe()` across all group members), a group's fetch position ended up past several thousand messages that were never delivered to the application. The messages were intact and consumable — other consumer groups on the same topic received all of them.

## Matching fixes in librdkafka 2.15.0

From the 2.15.0 changelog (all KIP-848 consumer fixes, #5541):

> - Defer the leave heartbeat until the assignment is revoked, so a member that exceeds `max.poll.interval.ms` rejoins cleanly instead of being rejected with a fatal `INVALID_REQUEST`.
> - Treat `GROUP_ID_NOT_FOUND` in the ConsumerGroupHeartbeat response as fatal, except while the member is already leaving.
> - Surface unexpected permanent broker-level heartbeat errors instead of retrying them in a loop.

The third item in particular matches the silent stalls we measured. librdkafka 2.14.2 also contains a relevant fix (SIGSEGV in `rd_kafka_cgrp_handle_LeaveGroup()` when the coordinator is unavailable during consumer close).

## Related (can file separately)

While debugging this we noticed `KafkaConsumerConsumeNum::Execute` (src/workers.cc) discards consumer error events — including `ERR__AUTO_OFFSET_RESET` — when they arrive after at least one regular message in the batch (`default:` case only calls `SetErrorBaton` when `m_messages.size() == eof_event_count`). This means offset-reset notifications never reach the JS layer, which made the skipped-offsets case above very hard to diagnose. Happy to open a separate issue for that if useful.

Thanks for maintaining this library!

Contributor guide

Open the contributing guide

Research direction

Start by locating the configuration and release entry points that bundle librdkafka for node-rdkafka 3.6.1; verify how the bundled version is selected and packaged for npm. Update the bundled dependency to librdkafka 2.15.0 or newer, validate the consumer fixes against the stated KIP-848 configuration, and publish an installable release.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
build-system, release, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.