rapidsai / rapidsai/dask-upstream-testing

Revert `distributed/tests/test_nanny.py::test_malloc_trim_threshold` skip

Open
#90 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
2
Forks
7
Avg merge
2m
Merged PRs (30d)
1

Description

This test is failing on main:

__________________________ test_malloc_trim_threshold __________________________

c = <Client: No scheduler connected>
s = <Scheduler 'tcp://127.0.0.1:38743', workers: 0, cores: 0, tasks: 0>
a = <Nanny: None, threads: 2>

    @pytest.mark.slow
    @pytest.mark.skipif(not LINUX, reason="Requires GNU libc")
    @gen_cluster(
        client=True,
        Worker=Nanny,
        nthreads=[("", 2)],
        worker_kwargs={"memory_limit": "2GiB"},
    )
    async def test_malloc_trim_threshold(c, s, a):
        """Test that the nanny sets the MALLOC_TRIM_THRESHOLD_ environment variable before
        starting the worker process.
    
        This test relies on these settings to work:
    
            distributed.nanny.pre-spawn-environ.MALLOC_TRIM_THRESHOLD_: 65536
            distributed.worker.multiprocessing-method: spawn
    
        We're deliberately not setting them explicitly in @gen_cluster above, as we want
        this test to trip if somebody changes distributed.yaml.
    
        Note
        ----
        This test may start failing in a future Python version if CPython switches to
        using mimalloc by default. If it does, a thorough benchmarking exercise is needed.
        """
        pytest.importorskip("numpy")
        da = pytest.importorskip("dask.array")
    
        arr = da.random.random(2**29 // 8, chunks="512 kiB")  # 0.5 GiB
        arr = c.persist(arr)
        await wait(arr)
        # Wait for heartbeat
        await async_poll_for(lambda: s.memory.process > 2**29, timeout=5)
        del arr
    
        # This is the delicate bit, as it relies on
        # 1. PyMem_Free() to be quick to invoke glibc free() when memory becomes available
        # 2. glibc free() to be quick to invoke the kernel's sbrk() when the same happens
        #
        # At the moment of writing, the readings are:
        # - 132        MiB after starting a new worker
        # - 670~725[1] MiB after arr has been fully computed
        # - 156~210[1] MiB at the end of this test, with MALLOC_TRIM_THRESHOLD_=65536
        # - 620~670[1] MiB at the end of this test, without MALLOC_TRIM_THRESHOLD_
        # [1] depends on distributed.scheduler.worker-saturation
>       await async_poll_for(lambda: s.memory.process < 300 * 2**20, timeout=5)

packages/distributed/distributed/tests/test_nanny.py:818: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <function test_malloc_trim_threshold.<locals>.<lambda> at 0x769972435300>
timeout = 5, fail_func = None, period = 0.05

    async def async_poll_for(predicate, timeout, fail_func=None, period=0.05):
        deadline = time() + timeout
        while not predicate():
            await asyncio.sleep(period)
            if time() > deadline:
                if fail_func is not None:
                    fail_func()
>               pytest.fail(f"condition not reached until {timeout} seconds")
E               Failed: condition not reached until 5 seconds

packages/distributed/distributed/utils_test.py:1257: Failed
----------------------------- Captured stderr call -----------------------------
2026-04-01 08:22:41,142 - distributed.scheduler - INFO - State start
2026-04-01 08:22:41,148 - distributed.scheduler - INFO -   Scheduler at:     tcp://127.0.0.1:38743
2026-04-01 08:22:41,149 - distributed.scheduler - INFO -   dashboard at:  http://127.0.0.1:38657/status
2026-04-01 08:22:41,150 - distributed.scheduler - INFO - Registering Worker plugin shuffle
2026-04-01 08:22:41,158 - distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:38899'
2026-04-01 08:22:42,050 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:34927
2026-04-01 08:22:42,050 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:34927
2026-04-01 08:22:42,050 - distributed.worker - INFO -           Worker name:                          0
2026-04-01 08:22:42,051 - distributed.worker - INFO -          dashboard at:            127.0.0.1:39669
2026-04-01 08:22:42,051 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:38743
2026-04-01 08:22:42,051 - distributed.worker - INFO - -------------------------------------------------
2026-04-01 08:22:42,051 - distributed.worker - INFO -               Threads:                          2
2026-04-01 08:22:42,051 - distributed.worker - INFO -                Memory:                   2.00 GiB
2026-04-01 08:22:42,051 - distributed.worker - INFO -       Local Directory: /tmp/dask-scratch-space/worker-gsz2u6xf
2026-04-01 08:22:42,051 - distributed.worker - INFO - -------------------------------------------------
2026-04-01 08:22:42,097 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:34927 name: 0
2026-04-01 08:22:42,100 - distributed.worker - INFO - Starting Worker plugin shuffle
2026-04-01 08:22:42,100 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:38743
2026-04-01 08:22:42,101 - distributed.worker - INFO - -------------------------------------------------
2026-04-01 08:22:42,099 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34927
2026-04-01 08:22:42,101 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46860
2026-04-01 08:22:42,102 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:38743
2026-04-01 08:22:42,143 - distributed.scheduler - INFO - Receive client connection: Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:42,145 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46870
2026-04-01 08:22:52,417 - distributed.scheduler - INFO - Remove client Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:52,419 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46870; closing.
2026-04-01 08:22:52,420 - distributed.scheduler - INFO - Remove client Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:52,423 - distributed.scheduler - INFO - Close client connection: Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:52,425 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:38899'. Reason: nanny-close
2026-04-01 08:22:52,426 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close
2026-04-01 08:22:52,429 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34927. Reason: nanny-close
2026-04-01 08:22:52,430 - distributed.worker - INFO - Removing Worker plugin shuffle
2026-04-01 08:22:52,433 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46860; closing.
2026-04-01 08:22:52,433 - distributed.core - INFO - Connection to tcp://127.0.0.1:38743 has been closed.
2026-04-01 08:22:52,434 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:34927 name: 0 (stimulus_id='handle-worker-cleanup-1775031772.4345043')
2026-04-01 08:22:52,436 - distributed.scheduler - INFO - Lost all workers
2026-04-01 08:22:52,441 - distributed.nanny - INFO - Worker closed
2026-04-01 08:22:53,012 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:38899' closed.
2026-04-01 08:22:53,014 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown
2026-04-01 08:22:53,017 - distributed.scheduler - INFO - Scheduler closing all comms

I didn't see it in distributed's own CI, passing or failing (it might be skipped?)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with distributed/tests/test_nanny.py::test_malloc_trim_threshold and run the targeted test on Linux with its numpy and dask.array imports. Read the referenced distributed.yaml settings for MALLOC_TRIM_THRESHOLD_ and the multiprocessing method, then compare the failure with the reported memory thresholds. Done means the skip is reverted only when the test reliably passes in the relevant CI environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.