Limited permission directories in tmpdirs don't get removed & show warnings

Context: I wanted to write a test to check behavior when permissions were not available, so turned to tmpdir with mkdir and chmod, assuming the temporary folder would be removed properly. However, admittedly at least in a rather extreme case of chmod(0), this leads to various warnings ending in:

...python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-07a6704d-839b-4882-8e94-9ae396c07a40/test_10
<class 'OSError'>: [Errno 39] Directory not empty: 'test_10'

I have tested this with pytest 6.1.0, though also 5.3.5.

The simplified test case, with error reproducible in a fresh venv with pytest 6.1.0 comes from something as simple as:

def test_1(tmpdir):
    tmpdir.mkdir("X").chmod(0)

Installed packages in the venv are:

Package            Version
------------------ -------
attrs              20.2.0 
importlib-metadata 2.0.0  
iniconfig          1.0.1  
packaging          20.4   
pip                18.1   
pkg-resources      0.0.0  
pluggy             0.13.1 
py                 1.9.0  
pyparsing          2.4.7  
pytest             6.1.0  
setuptools         40.8.0 
six                1.15.0 
toml               0.10.1 
zipp               3.2.0 

This is also confusing since it works fine 3 times, but once the cache of 3 old test runs is garbage collected, errors like the above are produced; in the simple venv case, something like as follows:

/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-bd1a01c6-2aa6-488b-a1a9-8729b0a89b3c/test_10
<class 'OSError'>: [Errno 39] Directory not empty: 'test_10'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-bd1a01c6-2aa6-488b-a1a9-8729b0a89b3c
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-neil/garbage-bd1a01c6-2aa6-488b-a1a9-8729b0a89b3c'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-b3a5e41f-5d44-4f3d-a147-f5b9cb0f3b3f/test_10
<class 'OSError'>: [Errno 39] Directory not empty: 'test_10'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-b3a5e41f-5d44-4f3d-a147-f5b9cb0f3b3f
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-neil/garbage-b3a5e41f-5d44-4f3d-a147-f5b9cb0f3b3f'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)

Reproduce the issue with the provided tmpdir test, then inspect _pytest/pathlib.py around rm_rf and trace the temporary-directory cleanup path. Done means the chmod(0) case no longer leaves the directory behind or emits the reported PytestWarning; add or update a focused regression test if the existing test layout supports it.