URLFrontier channels are always plaintext and the module has no TLS option
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
## What happens
`ManagedChannelUtil.createChannel()` builds every gRPC channel with `usePlaintext()`. It is the only channel factory in the module: `Spout`, `StatusUpdaterBolt` and `QueueRegulatorBolt` all go through it. There is no configuration key for TLS, mutual TLS or channel credentials anywhere in `external/urlfrontier`, so an operator who wants the link encrypted cannot get it, even deliberately.
## Where
`external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/ManagedChannelUtil.java:46-53`. Related config keys: `urlfrontier.address`, `urlfrontier.host`, `urlfrontier.port`.
```java
static ManagedChannel createChannel(@NotNull String address) {
...
return ManagedChannelBuilder.forTarget(address).usePlaintext().build();
}
```
## Why it matters
URLFrontier is typically a separate service, and the module supports several `urlfrontier.address` entries for exactly that layout, so the channel usually crosses a host boundary. Everything the crawler knows about a URL travels on it: the URL itself and its metadata, which can include cookies when cookie support is enabled. Anyone who can read that segment reads the crawl state, and anyone who can write to it can change what the workers fetch, because per-URL metadata influences request behaviour. Operators who need the link protected today have to tunnel it themselves.
## Reproduction
No automated test. Demonstrating the transport would need a running frontier service and a second gRPC endpoint with TLS, which is not something to put in a unit test. Manual steps:
1. `grep -rn "usePlaintext\|TlsChannelCredentials" external/urlfrontier/src/main` returns only line 52 of `ManagedChannelUtil.java` and no TLS builder.
2. `grep -rn "urlfrontier\." external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Constants.java` lists every configuration key the module reads. None of them concerns transport security.
3. Start a URLFrontier service configured to require TLS and point a topology at it. The channel fails to connect, and no setting changes that.
## Suggested fix
Add configuration to `ManagedChannelUtil.createChannel` for TLS channel credentials, built with `Grpc.newChannelBuilder(address, TlsChannelCredentials...)`, with optional client certificate and trust roots. Keep plaintext as the default for now so existing deployments keep working, and log a warning at channel creation when plaintext is used. Flipping the default to TLS is a breaking change and belongs in a major release, not in a patch.
Contributor guide
Research direction
Start with external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/ManagedChannelUtil.java:46-53 and review the configuration keys in Constants.java, then trace its use from Spout, StatusUpdaterBolt, and QueueRegulatorBolt. Add configurable TLS credentials, optional client certificate and trust roots, preserve plaintext as the default with a warning, and verify the configuration against a TLS-required URLFrontier service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100