[Bug]: ZMQ _send_recv raises SaltReqTimeoutError after hard-coded 300ms POLLOUT miss instead of configured request timeout
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
What happened?
In salt/transport/zeromq.py, both AsyncReqMessageClient._send_recv and RequestClient._send_recv treat a single hard-coded 300ms socket.poll(..., zmq.POLLOUT) miss as an immediate request timeout:
if not await socket.poll(300, zmq.POLLOUT):
if not future.done():
future.set_exception(
SaltReqTimeoutError("Socket not ready for sending")
)
if not self._closing:
await self._reconnect()
break
This produces the familiar minion log line:
Request timed out while waiting for a response. reconnecting.
even when the caller passed a much larger timeout to send() (e.g. via return_retry_timer / request_channel_timeout). That timeout is applied separately with io_loop.call_later(timeout, self._timeout_message, future), which is the intended request deadline.
Expected behavior
The 300ms value should only be a readiness poll slice (same pattern as the POLLIN loop that follows). _send_recv should keep polling for POLLOUT until either:
- the socket becomes write-ready and the message is sent, or
- the future is already done because
_timeout_messagefired at the configured request timeout.
It should not call future.set_exception(SaltReqTimeoutError(...)) on a POLLOUT miss before that configured deadline.
Contrast with POLLIN path (already correct)
while True:
if future.done():
break
ready = await socket.poll(300, zmq.POLLIN)
...
Suggested fix
Mirror POLLIN:
sent = False
while not future.done():
if await socket.poll(300, zmq.POLLOUT):
await socket.send(message)
sent = True
break
if not sent:
continue # configured timeout already completed the future
Impact
Under master backpressure / slow REQ socket write readiness (e.g. overloaded Salt master), job results can complete on the minion but fail to publish because the return path aborts after 300ms despite return_retry_timer / request_channel_timeout being configured much higher. Raising those options has no effect on this POLLOUT path because the 300ms constant is not read from opts.
Notes
- Observed on Salt
3008.0rc4(not in the Major version dropdown yet); code pattern is present in currentsalt/transport/zeromq.pyfor both request client classes. - 300ms is not configurable via any Salt opt (
zmq_backlog/pub_hwm/ etc. do not apply).
Type of salt install
Official rpm (appliance-packaged Salt)
Major version
- 3007.x (pattern also present on 3008.0rc4)
What supported OS are you seeing the problem on?
- photon-5 (vCenter / VCF appliance minion)
salt --versions-report output
Salt: 3008.0rc4
Python: 3.14.5
(Truncated; full report available on request.)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/transport/zeromq.py at AsyncReqMessageClient._send_recv and RequestClient._send_recv, comparing their POLLOUT handling with the existing POLLIN loop. Verify that a 300ms POLLOUT miss remains a readiness slice and does not complete the future before the configured request timeout. Done means both clients wait for write readiness or the configured timeout without raising prematurely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100