apache / apache/stormcrawler

fetcher.thread.timeout: cancel the okhttp call instead of interrupting a helper thread

Open
#2,134 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

`fetcher.thread.timeout` (#996, #1861) wraps `protocol.getProtocolOutput()` in a `Future` run on a single-thread executor owned by each `FetcherThread`, and cancels it with `future.cancel(true)` when the deadline passes.

Two problems with that:

- `cancel(true)` is a `Thread.interrupt()`, which okhttp does not honour while connecting or reading. The helper thread stays blocked on the socket until okhttp's own timeouts fire, and the `FetcherThread` queues behind it at its next fetch because the executor is single-threaded. One host that dribbles bytes can take a `FetcherThread` out of service for as long as `http.timeout`, or `topology.message.timeout.secs` (300s by default) which is the value the okhttp `callTimeout` is currently set from.
- It doubles the thread count of the bolt whenever the option is on: 50 `FetcherThread`s plus 50 `FetcherTimeout` helpers.

okhttp exposes the right primitive: `Call.timeout()` sets a per-call deadline enforced by okio's single shared watchdog thread, and on expiry the call is cancelled, the socket closed, and the fetching thread gets an `InterruptedIOException` immediately.

Proposal: `okhttp.HttpProtocol` applies `fetcher.thread.timeout` as a per-call deadline, and `Protocol` gains a `default boolean supportsFetchTimeout()` (false by default, true for okhttp when configured). `FetcherBolt` and `SimpleFetcherBolt` keep the `Future` path only for protocols that do not support it, creating the helper lazily. For the default okhttp protocol the fetch runs in the `FetcherThread` itself, the timeout is a real cancellation, and no helper threads exist.

Contributor guide

Open the contributing guide

Research direction

Start with okhttp.HttpProtocol, Protocol, FetcherBolt, and SimpleFetcherBolt, tracing how fetcher.thread.timeout currently creates and cancels the Future helper. Confirm the default okhttp path applies a per-call deadline and runs in FetcherThread, while unsupported protocols retain the lazy helper path and no extra helper threads are created for okhttp.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, networking
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.