test_mem_leak_in_event_listener flakes because job_not_running leaks tag_map entries
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
tests/pytests/integration/netapi/rest_tornado/test_minions_api_handler.py::test_mem_leak_in_event_listener intermittently fails with AssertionError: assert 2 == 0 for app.event_listener.tag_map.
Observed on the 3008.x nightly (Amazon Linux 2 integration zeromq 4, run 32913591224 job 98023096704) — first attempt fails with two salt/job/<jid>/ret prefix-matcher futures left in tag_map, retry passes. Same pattern reproduced on merge-forward PR #70132.
Root cause: saltnado.py::_disbatch_local spawns job_not_running via ioloop.spawn_callback to ping saltutil.find_job. When the outer job's is_finished completes first, job_not_running calls event.set_result(None) on the in-flight ping future but does not remove it from event_listener.tag_map / timeout_map. Because spawn_callback runs independently of the handler, the ping may be registered after on_finish -> clean_by_request has already emptied request_map for the handler, so the leaked entry only clears when the gather_job_timeout callback fires (10 s default, 30 s under the netapi test fixture).
The test only allows a 1 s wait before asserting the maps are empty, so any leaked ping trips the assertion.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/pytests/integration/netapi/rest_tornado/test_minions_api_handler.py::test_mem_leak_in_event_listener and inspect saltnado.py::_disbatch_local, especially the job_not_running callback and event-listener cleanup. Reproduce the flaky assertion and trace tag_map, timeout_map, and request_map around on_finish. Done means the test reliably observes empty maps within its 1-second wait without relying on the timeout callback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100