saltstack / saltstack/salt

[Bug]: ZMQ _send_recv raises SaltReqTimeoutError after hard-coded 300ms POLLOUT miss instead of configured request timeout

Open
#69,802 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

In salt/transport/zeromq.py, both AsyncReqMessageClient._send_recv and RequestClient._send_recv treat a single hard-coded 300ms socket.poll(..., zmq.POLLOUT) miss as an immediate request timeout:

if not await socket.poll(300, zmq.POLLOUT):
    if not future.done():
        future.set_exception(
            SaltReqTimeoutError("Socket not ready for sending")
        )
    if not self._closing:
        await self._reconnect()
    break

This produces the familiar minion log line:

Request timed out while waiting for a response. reconnecting.

even when the caller passed a much larger timeout to send() (e.g. via return_retry_timer / request_channel_timeout). That timeout is applied separately with io_loop.call_later(timeout, self._timeout_message, future), which is the intended request deadline.

Expected behavior

The 300ms value should only be a readiness poll slice (same pattern as the POLLIN loop that follows). _send_recv should keep polling for POLLOUT until either:

  1. the socket becomes write-ready and the message is sent, or
  2. the future is already done because _timeout_message fired at the configured request timeout.

It should not call future.set_exception(SaltReqTimeoutError(...)) on a POLLOUT miss before that configured deadline.

Contrast with POLLIN path (already correct)

while True:
    if future.done():
        break
    ready = await socket.poll(300, zmq.POLLIN)
    ...

Suggested fix

Mirror POLLIN:

sent = False
while not future.done():
    if await socket.poll(300, zmq.POLLOUT):
        await socket.send(message)
        sent = True
        break
if not sent:
    continue  # configured timeout already completed the future

Impact

Under master backpressure / slow REQ socket write readiness (e.g. overloaded Salt master), job results can complete on the minion but fail to publish because the return path aborts after 300ms despite return_retry_timer / request_channel_timeout being configured much higher. Raising those options has no effect on this POLLOUT path because the 300ms constant is not read from opts.

Notes

  • Observed on Salt 3008.0rc4 (not in the Major version dropdown yet); code pattern is present in current salt/transport/zeromq.py for both request client classes.
  • 300ms is not configurable via any Salt opt (zmq_backlog / pub_hwm / etc. do not apply).
Type of salt install

Official rpm (appliance-packaged Salt)

Major version
  • 3007.x (pattern also present on 3008.0rc4)
What supported OS are you seeing the problem on?
  • photon-5 (vCenter / VCF appliance minion)
salt --versions-report output
Salt: 3008.0rc4
Python: 3.14.5

(Truncated; full report available on request.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/transport/zeromq.py at AsyncReqMessageClient._send_recv and RequestClient._send_recv, comparing their POLLOUT handling with the existing POLLIN loop. Verify that a 300ms POLLOUT miss remains a readiness slice and does not complete the future before the configured request timeout. Done means both clients wait for write readiness or the configured timeout without raising prematurely.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.