confluentinc / confluentinc/confluent-kafka-javascript
eachBatch: fatal/unrecoverable consumer errors (e.g. ERR__STATE) are swallowed — consumer wedges silently with no crash event or run() rejection
- Dominant language
- TypeScript
- Stars
- 304
- Forks
- 45
- Avg merge
- 11h 47m
- Merged PRs (30d)
- 5
Description
## Summary
When using the KafkaJS-compatible `eachBatch` consumer, there is no supported way to observe that the consumer has entered an **unrecoverable/fatal state**. librdkafka errors that occur in the background consume loop (e.g. repeated `ERR__STATE`) are caught internally, logged, and retried forever. The consumer stops making progress (no messages delivered, no offsets committed), but:
- the `run()` promise does **not** reject,
- the `eachBatch` callback is **not** invoked with an error,
- no event is emitted (`consumer.on(...)` / `consumer.events` are `notImplemented()`),
- the Node process stays alive and "healthy" from the OS/orchestrator point of view.
Net effect: the consumer **silently wedges**. Under an orchestrator that only restarts a task on process exit (ECS/K8s without an app-level liveness probe), such a task lingers indefinitely, consuming CPU but doing no work, until manually recycled.
## Environment
- `@confluentinc/kafka-javascript`: **1.9.0**
- API: KafkaJS-compatible (`new Kafka(...).consumer(...)`, `consumer.run({ eachBatch })`)
- Node.js 24, Linux/arm64
## What we observe (logs)
The only signal is library-level logging from the binding (facility `BINDING`). Right before a consumer goes silent we repeatedly see ERR__STATE-family messages:
```
Consumer encountered error while consuming. Retrying. Error details: KafkaJSError: Seek can only be called while connected. : KafkaJSError: Seek can only be called while connected.
at Consumer.seek (.../@confluentinc/kafka-javascript/lib/kafkajs/_consumer.js:1909:13)
at #batchProcessor (.../_consumer.js:1451:12)
...
```
```
Consumer encountered error while storing offset. Error details: Error: Local: Erroneous state:Error: Local: Erroneous state
at KafkaConsumer._offsetsStoreSingle (.../lib/kafka-consumer.js:752:15)
...
```
```
Could not get watermark offsets for batch: Error: Local: Erroneous state
```
After these, the instance stops emitting any `eachBatch` activity and never recovers — but the process keeps running.
## Where this happens in the code (v1.9.0, `lib/kafkajs/_consumer.js`)
1. **Background worker swallows all consume/processing errors and retries forever.** In `#worker` the `try/catch` around the fetch + per-message processor logs and continues the loop:
```js
} catch (e) {
/* Since this error cannot be exposed to the user in the current situation, just log and retry.
* This is due to restartOnFailure being set to always true. */
if (this.#logger)
this.#logger.error(`Consumer encountered error while consuming. Retrying. Error details: ${e} : ${e.stack}`, ...);
}
```
(around `_consumer.js:1533-1537`). There is no way to provide a `retry.restartOnFailure`-style callback (as in vanilla KafkaJS) to make the consumer stop on fatal errors.
2. **Fatal `event.error` is only logged after connect.** `#errorCb` — *"Callback for the event.error event, either fails the initial connect(), or logs the error."* (`_consumer.js:768`):
```js
#errorCb(err) {
if (this.#state < ConsumerState.CONNECTED) {
if (!this.#connectionError) this.#connectionError = err;
} else {
this.#logger.error(err, ...); // <- only logged; not surfaced
}
}
```
3. **The KafkaJS event API is not implemented**, so the usual `consumer.on(consumer.events.CRASH, ...)` escape hatch is unavailable:
```js
on(/* eventName, listener */) { notImplemented(); } // _consumer.js:2086
get events() { notImplemented(); return null; } // _consumer.js:2104
```
In vanilla KafkaJS, an unrecoverable consumer error eventually emits a `consumer.crash` event (with `restart: false` once retries are exhausted), which apps use to fail fast and let the orchestrator restart them. With this library there appears to be **no supported equivalent**.
## Expected behavior
A supported way for application code to detect a fatal/unrecoverable consumer state and react (typically: exit and let the orchestrator restart). Any one of these would solve it:
- Implement `consumer.on(consumer.events.CRASH, ...)` (and/or `DISCONNECT`) for the KafkaJS-compat consumer; emit `CRASH` when the consumer can no longer make progress.
- Honor a `retry.restartOnFailure(error) => boolean` callback in `run(...)`, so the app can opt to stop on fatal errors (matching KafkaJS semantics).
- Reject the `run()` promise (or surface via `eachBatch`) on fatal/unrecoverable `event.error` instead of only logging.
## Current workaround
Because there is no public API, we reach into the **private** internal node-rdkafka client via `consumer._getInternalClient()` (available after `connect()`) and subscribe to its events:
- `event.error` with `err.isFatal === true` (or `ERR__FATAL`) → treat as fatal immediately.
- `ERR__STATE` (the `Seek can only be called while connected` / `Local: Erroneous state` family) → arm a ~45s timer; cancel it on any liveness signal: `rebalance`, `offset.commit`, `partition.eof` (we set `enable.partition.eof: true`), or a delivered batch. If the timer fires (no recovery), we emit our own crash signal.
On crash we run graceful shutdown and `process.exit`, so the orchestrator replaces the task. This works but relies on a private method (`_getInternalClient`) and on string/code matching of librdkafka errors, which is fragile across versions — hence this request for a first-class API.
## Repro sketch
1. Start an `eachBatch` consumer.
2. Drive the underlying client into a persistent `ERR__STATE` (e.g. broker connectivity loss / forced state error during a rebalance while seeking/committing).
3. Observe: the `Consumer encountered error while consuming. Retrying.` / `Erroneous state` logs repeat, the consumer stops delivering batches, **but** `run()` never rejects, `eachBatch` is never called with an error, and no event fires. The process stays up indefinitely.
Happy to provide more detail or test a fix. Thanks!
Contributor guide
Research direction
Start in lib/kafkajs/_consumer.js by reading #worker, #errorCb, run(), and the unimplemented on() and events accessors. Trace how consume errors are retried and how fatal event.error values are handled. Done means the KafkaJS-compatible consumer exposes a documented application-level fatal-state signal or stop policy instead of silently retrying forever.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100