core: a wedged DNS lookup can hold push shutdown past its drain budget
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 20
- Forks
- 2
- Avg merge
- 5h 10m
- Merged PRs (30d)
- 127
Description
Follow-up to #843 / #844, merged in #943. Documented on PinnedResolver rather than fixed there, because both remedies reach past the push path.
The bound
PinnedResolver::resolve (crates/flexiq-core/src/http/resolver.rs) runs getaddrinfo through tokio::task::spawn_blocking, because the call blocks and must not sit on an async worker. Once a blocking task has started, tokio cannot cancel it — dropping the future that awaits it detaches the caller, not the work.
crates/flexiq-core/src/worker/runner.rs builds the worker's multi-thread runtime and lets it drop at the end of the thread closure, with no shutdown_timeout and no shutdown_background. Dropping a runtime waits for outstanding blocking tasks.
So a wedged DNS lookup holds process shutdown open past HttpTargetConfig::shutdown_drain (Helm: push.drain). The chart now sizes terminationGracePeriodSeconds from 2 × push.drain, and a lookup outliving that window is SIGKILLed mid-shutdown — every lease still open at that instant falls to the stale-job reaper instead of settling.
What it is not
Worth stating, because it bounds the priority:
- Not unbounded.
getaddrinfois limited by the platform resolver's own configuration —resolv.conf'stimeout×attemptsacross each nameserver. The default is single-digit seconds; a pathological one is tens. - Not fixable with a timeout on the resolution future. The blocking task keeps running regardless. That would convert a slow lookup into a fast error while leaving the shutdown wait exactly as it is.
- Not push-specific in its remedy. The runtime is shared by every dispatcher.
The two candidate remedies
1. A cancellable async resolver. Removes the blocking pool from the path entirely. Cost is a DNS crate: this feature was built to add zero new nodes to Cargo.lock, and the workspace MSRV floor is 1.88, gated in CI by publish-crates.yml. That floor is the same constraint that forced SigV4 to be hand-rolled here rather than taking aws-sigv4 (which declares 1.94.1). Any candidate needs checking against both.
2. A bounded runtime shutdown. Runtime::shutdown_timeout in runner.rs. Cheap to write, but it changes shutdown semantics for attach and native workers too, to bound something only the push path can currently trigger. Needs a decision about what the right bound is for a worker in general, not just for a dispatch whose drain budget is already configured.
Neither is obviously correct, which is why this is an issue rather than a patch.
Acceptance
A wedged name lookup at shutdown does not extend the process's exit past the configured drain budget, and the fix does not regress attach or native shutdown. Whichever route is taken, the note on PinnedResolver should be updated or removed so it does not outlive the limitation it records.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with PinnedResolver::resolve in crates/flexiq-core/src/http/resolver.rs and the runtime construction and drop path in crates/flexiq-core/src/worker/runner.rs. Compare the cancellable resolver and bounded shutdown options against the MSRV and zero-new-Cargo.lock constraints, then verify that a wedged lookup cannot exceed the configured drain budget without regressing attach or native shutdown; update the PinnedResolver note afterward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100