[Bug]:KafkaIO OAUTHBEARER auth against GCP Managed Service for Apache Kafka broken starting 2.69.0 (works on 2.68.0)
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
KafkaIO OAUTHBEARER auth against GCP Managed Service for Apache Kafka broken starting 2.69.0 (works on 2.68.0)
Using Python `ReadFromKafka` with `sasl.mechanism=OAUTHBEARER` and
`sasl.login.callback.handler.class=com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler`
to connect to **GCP Managed Service for Apache Kafka** from a Dataflow streaming job
(`DataflowRunner`, `--experiments=use_runner_v2`), the Kafka consumer fails 100% of
connection attempts with:
```
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
at org.apache.beam.sdk.io.kafka.KafkaIO$Read$GenerateKafkaSourceDescriptor...
```
with worker logs showing, for every connection attempt:
```
[Consumer clientId=..., groupId=...] Connection to node -1 (:9092) terminated
during authentication. This may happen due to any of the following reasons:
(1) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic),
(2) Transient network issue.
```
This is **not** a network/firewall issue — same VPC/subnetwork/service-account/broker
worked correctly on Beam 2.68.0 and earlier.
## Bisection result
Tested directly against a live GCP Managed Kafka cluster (Dataflow, `DataflowRunner`,
identical pipeline code, identical JDK build (`17.0.18+8-Debian-1deb12u1`, controlled via
a pinned custom SDK container image so the JDK is not a variable), identical
`kafka-clients` (3.9.0) and identical `GcpLoginCallbackHandler.class` bytecode
(SHA256-verified) across every version tested:
| Beam version | Result |
|---|---|
| 2.67.0 | ✅ Works |
| 2.68.0 | ✅ Works (confirmed live data flowing) |
| **2.69.0** | ❌ **Fails** — 100% auth termination |
| 2.71.0 | ❌ Fails |
| 2.75.0 | ❌ Fails |
The break happens exactly between the **2.68.0 → 2.69.0** release.
## Dependency diff, 2.68.0 → 2.69.0
Diffing every `META-INF/maven/*/pom.properties` entry bundled in
`beam-sdks-java-io-expansion-service-{2.68.0,2.69.0}.jar`, after excluding
unrelated file-format IO connectors (ORC/Parquet/JTS, bundled in the same
shadowJar but irrelevant to Kafka), the changes touching the HTTP/auth path are:
```
org.apache.httpcomponents.client5:httpclient5 5.4.3 -> 5.5
io.confluent:kafka-schema-registry-client 7.6.1 -> 7.5.5 (downgrade)
io.confluent:common-utils 7.6.1 -> 7.5.5 (downgrade)
net.bytebuddy:byte-buddy 1.14.12 -> 1.17.7
```
`google-auth-library-oauth2-http` and `google-auth-library-credentials` are
**unchanged** across this specific boundary (both `1.37.1` / `1.30.0`), so this
looks like a transport-layer (`httpclient5`) or Confluent schema-registry-client
regression rather than a `google-auth-library` version bump, despite
`google-auth-library-oauth2-http` moving significantly (1.36.0 → 1.47.0) across
the wider 2.67.0 → 2.75.0 range.
## What's been ruled out
- `kafka-clients` version: identical (3.9.0) at every version tested
- `GcpLoginCallbackHandler.class`: byte-identical (SHA256-verified) at every version tested
- `managed-kafka-auth-login-handler` (Google's own artifact): identical, `1.0.5`
- JDK patch version: identical (`17.0.18+8-Debian-1deb12u1`), controlled via a pinned
custom SDK container image built via multi-stage `COPY` from a known-good image,
ruling out JDK/TLS-default drift as a factor
- Network/firewall/subnetwork/service account/IAM: identical config across all tests;
ruled out because the *same* infra with only the Beam SDK version changed reproduces
the failure deterministically
## Environment
- Runner: `DataflowRunner`, `--experiments=use_runner_v2`,
`--dataflow_service_options=streaming_mode_at_least_once`
- SDK: Python 3.12, `apache-beam[gcp]`
- Kafka: GCP Managed Service for Apache Kafka, `SASL_SSL` + `OAUTHBEARER`
- Auth: `com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler` (Application
Default Credentials via the pipeline's attached service account)
## Reproduction
Minimal `consumer_config` used:
```python
{
"bootstrap.servers": ":9092",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "OAUTHBEARER",
"sasl.login.callback.handler.class": "com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler",
"sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;",
}
```
Passed to `ReadFromKafka(consumer_config=consumer_config, topics=[...])` in a
streaming pipeline submitted with `--sdk_container_image` pinned to a custom image
per Beam version tested (JDK controlled/identical across all).
Happy to share the full bisection harness (build + throwaway-consumer-group test
script) if useful for reproducing this quickly across future release candidates.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start by reproducing the minimal Python ReadFromKafka pipeline with the stated consumer_config against the 2.68.0 and 2.69.0 SDK containers. Inspect KafkaIO$Read$GenerateKafkaSourceDescriptor and compare the expansion-service dependency contents, especially httpclient5 and the Confluent clients. Done means OAUTHBEARER authentication succeeds against GCP Managed Kafka on the affected Beam versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, java, kafka, python
- Domain
- authentication, cloud, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100