[BUG] filelist's findall() should prefer original over links

Start with setuptools._distutils.filelist.findall and the UniqueFolder filter identified in the report. Run the provided temporary-directory reproduction, checking how os.walk ordering and linked directories affect the returned paths. Done means the original A, B, and C folders are included consistently, while symlinked duplicates are excluded.

setuptools version

62.1.0

Python version

3.8.13

OS

Windows 10

Additional environment information

No response

Description

When trying to package an sdist of a library I maintain, certain folders were no longer being included when using a recent version of setuptools. Stepping through the code with PDB, I realized that the UniqueFolder filter was excluding them (introduced in https://github.com/pypa/setuptools/pull/2714). For context, this is the kind of folder setup in use:

folderA/links/[symlink to folderB]
folderA/links/[symlink to folderC]
folderB/links/[symlink to folderC]
folderC

The problem is that instead of including folderB and folderC, only the links in folderA/links is included.

For me, these links gets created as part of some Javascript assets that I build an distribute with my package (https://github.com/vidartf/ipydatawidgets). The workarounds I can see:

• Rename the folders so that os.walk traverses them in the "right" order. This is brittle as the docs for os.walk says "Whether or not the lists are sorted depends on the file system." This is another argument for why the current implementation is also problematic: The paths of the files that get included in your build are dependent on the OS implementation details!
• Add a separate build step that unlinks these folders before building, and then re-links them afterwards.

Expected behavior

The expectation would be that the original folders (folderA, folderB and folderC) are all included, and if anything the symlinks pointing to them are excluded.

How to Reproduce

import tempfile
import os
import pathlib
from setuptools._distutils.filelist import findall

with tempfile.TemporaryDirectory() as root:
    d = pathlib.Path(root)
    os.mkdir(d / "A")
    os.mkdir(d / "B")
    os.mkdir(d / "C")
    (d / "A" / "fileinA.txt").write_text("foo")
    (d / "B" / "fileinB.txt").write_text("foo")
    (d / "C" / "fileinC.txt").write_text("foo")
    # Note: This will work if the sort order is changed!
    os.symlink(d / "C", d / "B" / "linkC", target_is_directory=True)
    os.symlink(d / "C", d / "A" / "linkC", target_is_directory=True)
    os.symlink(d / "B", d / "A" / "linkB", target_is_directory=True)

    files = findall(d)
    print(files)

Output

[
'<tmp root>\\tmpjswv9fx2\\A\\fileinA.txt',
'<tmp root>\\tmpjswv9fx2\\A\\linkB\\fileinB.txt',
'<tmp root>\\tmpjswv9fx2\\A\\linkB\\linkC\\fileinC.txt'
]