saltstack / saltstack/salt

test_mem_leak_in_event_listener flakes because job_not_running leaks tag_map entries

Open
#70,152 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

tests/pytests/integration/netapi/rest_tornado/test_minions_api_handler.py::test_mem_leak_in_event_listener intermittently fails with AssertionError: assert 2 == 0 for app.event_listener.tag_map.

Observed on the 3008.x nightly (Amazon Linux 2 integration zeromq 4, run 32913591224 job 98023096704) — first attempt fails with two salt/job/<jid>/ret prefix-matcher futures left in tag_map, retry passes. Same pattern reproduced on merge-forward PR #70132.

Root cause: saltnado.py::_disbatch_local spawns job_not_running via ioloop.spawn_callback to ping saltutil.find_job. When the outer job's is_finished completes first, job_not_running calls event.set_result(None) on the in-flight ping future but does not remove it from event_listener.tag_map / timeout_map. Because spawn_callback runs independently of the handler, the ping may be registered after on_finish -> clean_by_request has already emptied request_map for the handler, so the leaked entry only clears when the gather_job_timeout callback fires (10 s default, 30 s under the netapi test fixture).

The test only allows a 1 s wait before asserting the maps are empty, so any leaked ping trips the assertion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/pytests/integration/netapi/rest_tornado/test_minions_api_handler.py::test_mem_leak_in_event_listener and inspect saltnado.py::_disbatch_local, especially the job_not_running callback and event-listener cleanup. Reproduce the flaky assertion and trace tag_map, timeout_map, and request_map around on_finish. Done means the test reliably observes empty maps within its 1-second wait without relying on the timeout callback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.