apache / apache/pulsar-connectors
[Bug] Canal source client is incompatible with modern protobuf-java (NoSuchMethodError on connect)
- Dominant language
- Java
- Stars
- 26
- Forks
- 25
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 1
Description
## Description
The Canal source connector's client library (`com.alibaba.otter:canal.client:1.1.7` / `canal.protocol:1.1.7`) is incompatible with the protobuf-java version the build resolves. Its generated `CanalPacket` code calls `com.google.protobuf.GeneratedMessageV3.makeExtensionsImmutable()`, a method **removed** in modern protobuf-java (the 3.21+/4.x line). So `CanalConnector.connect()` throws `NoSuchMethodError` and the source can never complete the canal-server handshake.
Found while adding the first integration test for the module (#48 / #98) — the module had zero tests, so this was never exercised.
## Evidence
On the canal module's **test** classpath, protobuf-java resolves to 4.31.1 (pulled by `pulsar-client`/`pulsar-functions-instance`), and `GeneratedMessageV3` in that version has no `makeExtensionsImmutable`. The integration test reproduces the `NoSuchMethodError` on `connect()` without a version pin.
## Nuance worth a maintainer's eye — is production affected?
- The module's **runtimeClasspath** (what the NAR bundles) resolves **no** external `protobuf-java` at all, and `canal.protocol-1.1.7.jar` does not bundle protobuf classes. So the NAR appears to rely on whatever protobuf the Pulsar runtime provides — which is modern — meaning the connector would very likely hit the same `NoSuchMethodError` **in production**, not just in the test. This should be confirmed against a real deployment.
## The fix in #98, and why it needs review
#98 makes the integration test pass by forcing `protobuf-java` to **3.6.1** (canal's declared version) via `configurations.all { resolutionStrategy { force(...) } }`. That works, but:
1. **3.6.1 is from 2018** and carries known CVEs (e.g. CVE-2021-22569, CVE-2022-3171). Pinning the whole module — including the shipped NAR — to it is heavy.
2. The demonstrated breakage is on the **test** classpath; whether the NAR needs the pin depends on the production question above. If production is unaffected, the force should be scoped to test configurations only.
3. **`canal.client` 1.1.8 exists** (latest). If it is built against a newer protobuf, upgrading is a cleaner fix than downgrading protobuf. Worth checking before committing to the 3.6.1 pin.
## Suggested resolution
Decide among: (a) upgrade to `canal.client`/`canal.protocol` 1.1.8 if protobuf-4-compatible; (b) shade/relocate protobuf for the canal NAR; or (c) keep the 3.6.1 pin but scope it correctly (test-only if production is unaffected) and accept the CVE tradeoff. #98 currently takes (c) module-wide; flagging for a decision.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the Canal source module's dependency configuration and inspect integration test #98, which reproduces the failure at CanalConnector.connect(). Compare the test and runtime classpaths, check whether canal.client/canal.protocol 1.1.8 is compatible, and confirm done when the handshake succeeds without an unsafe module-wide dependency choice.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100