confluentinc / confluentinc/confluent-kafka-javascript

eachBatch: fatal/unrecoverable consumer errors (e.g. ERR__STATE) are swallowed — consumer wedges silently with no crash event or run() rejection

Open
#497 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

## Summary

When using the KafkaJS-compatible `eachBatch` consumer, there is no supported way to observe that the consumer has entered an **unrecoverable/fatal state**. librdkafka errors that occur in the background consume loop (e.g. repeated `ERR__STATE`) are caught internally, logged, and retried forever. The consumer stops making progress (no messages delivered, no offsets committed), but:

- the `run()` promise does **not** reject,
- the `eachBatch` callback is **not** invoked with an error,
- no event is emitted (`consumer.on(...)` / `consumer.events` are `notImplemented()`),
- the Node process stays alive and "healthy" from the OS/orchestrator point of view.

Net effect: the consumer **silently wedges**. Under an orchestrator that only restarts a task on process exit (ECS/K8s without an app-level liveness probe), such a task lingers indefinitely, consuming CPU but doing no work, until manually recycled.

## Environment

- `@confluentinc/kafka-javascript`: **1.9.0**
- API: KafkaJS-compatible (`new Kafka(...).consumer(...)`, `consumer.run({ eachBatch })`)
- Node.js 24, Linux/arm64

## What we observe (logs)

The only signal is library-level logging from the binding (facility `BINDING`). Right before a consumer goes silent we repeatedly see ERR__STATE-family messages:

```
Consumer encountered error while consuming. Retrying. Error details: KafkaJSError: Seek can only be called while connected. : KafkaJSError: Seek can only be called while connected.
at Consumer.seek (.../@confluentinc/kafka-javascript/lib/kafkajs/_consumer.js:1909:13)
at #batchProcessor (.../_consumer.js:1451:12)
...
```

```
Consumer encountered error while storing offset. Error details: Error: Local: Erroneous state:Error: Local: Erroneous state
at KafkaConsumer._offsetsStoreSingle (.../lib/kafka-consumer.js:752:15)
...
```

```
Could not get watermark offsets for batch: Error: Local: Erroneous state
```

After these, the instance stops emitting any `eachBatch` activity and never recovers — but the process keeps running.

## Where this happens in the code (v1.9.0, `lib/kafkajs/_consumer.js`)

1. **Background worker swallows all consume/processing errors and retries forever.** In `#worker` the `try/catch` around the fetch + per-message processor logs and continues the loop:
```js
} catch (e) {
/* Since this error cannot be exposed to the user in the current situation, just log and retry.
* This is due to restartOnFailure being set to always true. */
if (this.#logger)
this.#logger.error(`Consumer encountered error while consuming. Retrying. Error details: ${e} : ${e.stack}`, ...);
}
```
(around `_consumer.js:1533-1537`). There is no way to provide a `retry.restartOnFailure`-style callback (as in vanilla KafkaJS) to make the consumer stop on fatal errors.

2. **Fatal `event.error` is only logged after connect.** `#errorCb` — *"Callback for the event.error event, either fails the initial connect(), or logs the error."* (`_consumer.js:768`):
```js
#errorCb(err) {
if (this.#state < ConsumerState.CONNECTED) {
if (!this.#connectionError) this.#connectionError = err;
} else {
this.#logger.error(err, ...); // <- only logged; not surfaced
}
}
```

3. **The KafkaJS event API is not implemented**, so the usual `consumer.on(consumer.events.CRASH, ...)` escape hatch is unavailable:
```js
on(/* eventName, listener */) { notImplemented(); } // _consumer.js:2086
get events() { notImplemented(); return null; } // _consumer.js:2104
```

In vanilla KafkaJS, an unrecoverable consumer error eventually emits a `consumer.crash` event (with `restart: false` once retries are exhausted), which apps use to fail fast and let the orchestrator restart them. With this library there appears to be **no supported equivalent**.

## Expected behavior

A supported way for application code to detect a fatal/unrecoverable consumer state and react (typically: exit and let the orchestrator restart). Any one of these would solve it:

- Implement `consumer.on(consumer.events.CRASH, ...)` (and/or `DISCONNECT`) for the KafkaJS-compat consumer; emit `CRASH` when the consumer can no longer make progress.
- Honor a `retry.restartOnFailure(error) => boolean` callback in `run(...)`, so the app can opt to stop on fatal errors (matching KafkaJS semantics).
- Reject the `run()` promise (or surface via `eachBatch`) on fatal/unrecoverable `event.error` instead of only logging.

## Current workaround

Because there is no public API, we reach into the **private** internal node-rdkafka client via `consumer._getInternalClient()` (available after `connect()`) and subscribe to its events:

- `event.error` with `err.isFatal === true` (or `ERR__FATAL`) → treat as fatal immediately.
- `ERR__STATE` (the `Seek can only be called while connected` / `Local: Erroneous state` family) → arm a ~45s timer; cancel it on any liveness signal: `rebalance`, `offset.commit`, `partition.eof` (we set `enable.partition.eof: true`), or a delivered batch. If the timer fires (no recovery), we emit our own crash signal.

On crash we run graceful shutdown and `process.exit`, so the orchestrator replaces the task. This works but relies on a private method (`_getInternalClient`) and on string/code matching of librdkafka errors, which is fragile across versions — hence this request for a first-class API.

## Repro sketch

1. Start an `eachBatch` consumer.
2. Drive the underlying client into a persistent `ERR__STATE` (e.g. broker connectivity loss / forced state error during a rebalance while seeking/committing).
3. Observe: the `Consumer encountered error while consuming. Retrying.` / `Erroneous state` logs repeat, the consumer stops delivering batches, **but** `run()` never rejects, `eachBatch` is never called with an error, and no event fires. The process stays up indefinitely.

Happy to provide more detail or test a fix. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start in lib/kafkajs/_consumer.js by reading #worker, #errorCb, run(), and the unimplemented on() and events accessors. Trace how consume errors are retried and how fatal event.error values are handled. Done means the KafkaJS-compatible consumer exposes a documented application-level fatal-state signal or stop policy instead of silently retrying forever.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.