bpm-crafters / bpm-crafters/process-engine-adapters-operaton

[c7#226] Kafka Producer creation on worker threads fails with "No LoginModule found for OAuthBearerLoginModule" due to wrong Thread Context ClassLoader

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Source: c7
Dominant language
Kotlin
Stars
1
Forks
0
Avg merge
11m
Merged PRs (30d)
3

Description

> Synced from https://github.com/bpm-crafters/process-engine-adapters-camunda-7/issues/226

### Steps to reproduce

* Library version: 2025.11.1
* JDK version: 17

The failure requires **three conditions to be met simultaneously**:

1. **Spring-Boot Fat-Jar** — `kafka-clients` lives in `BOOT-INF/lib` and is only visible via the Spring-Boot `LaunchedClassLoader`, not via arbitrary worker classloaders.
2. **SASL/OAUTHBEARER** — the JDK `LoginContext` dynamically loads the configured JAAS `LoginModule` via the Thread Context ClassLoader (TCCL).
3. **Lazy Producer creation on a foreign thread** — the first `KafkaTemplate.send(...)` happens on a bpm-crafters worker thread whose TCCL is not the `LaunchedClassLoader`.

Concrete flow leading to the failure:

1. A bpm-crafters `process-engine-worker` processes an external Camunda task.
2. The handler triggers a send via `KafkaTemplate`.
3. `DefaultKafkaProducerFactory` creates the `KafkaProducer` lazily on this first `send()`.
4. Kafka initializes SASL/OAUTHBEARER and the JAAS `LoginContext`.
5. The `LoginContext` tries to load `OAuthBearerLoginModule` via the worker thread's TCCL.
6. That classloader cannot see the nested `kafka-clients` classes in `BOOT-INF/lib`, so producer creation fails.

Minimal reproducible:

[See test in PR](https://github.com/bpm-crafters/process-engine-adapters-camunda-7/pull/227/changes#r3747757359)

### Expected behaviour

The Kafka producer should be created successfully regardless of which thread triggers the first `send()`. Libraries such as the bpm-crafters adapter that execute foreign user code on their own worker threads should set the TCCL to a sensible ClassLoader (typically the one that loaded the application/handler) or offer a configurable `ThreadFactory`, so that SASL/OAUTHBEARER JAAS module loading works reliably.

### Actual behaviour

On the first Kafka send from a worker thread, producer construction aborts with `Failed to construct kafka producer`. The root cause in the exception chain is:

```
javax.security.auth.login.LoginException:
No LoginModule found for
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
```

The error is **intermittent** and often appears only once during a JVM's lifetime. This is explained by Kafka's static, JVM-wide `LoginManager` cache (keyed among other things by `sasl.jaas.config`):

* **First attempt:** a thread with an unsuitable TCCL initializes first, the `LoginModule` is not found, and no successful cache entry is created.
* **Later attempt:** a retry or listener runs on a thread with the correct `LaunchedClassLoader`, the `LoginManager` is created successfully and cached.
* **Afterwards:** further threads reuse the cached `LoginManager`, so the error does not reoccur even though the underlying classloader difference persists.

This is effectively a **race condition** — whichever thread initializes the JAAS configuration successfully first determines the observed behaviour. Camunda retries can mask the failure, but they can also be fully exhausted and thereby cause an incident.

### Suggested fix

https://github.com/bpm-crafters/process-engine-adapters-camunda-7/pull/227

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the minimal reproducible test linked in PR 227 and trace the KafkaTemplate.send path through DefaultKafkaProducerFactory. Check how the worker thread's TCCL is established during lazy producer creation. Done means the producer is created successfully from a foreign worker thread and the reproducer passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, kotlin, spring-boot
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.