Proposal to change implementation of globbing for possible performance increase

Hello,
I am using the build-in SFTP implementation to access an SFTP server. Unfortunately, this server is pretty slow, so I used globbing with the hope it will speed up things.

The files on this server are ordered in different subdirectories. Only files in all subdirs that start with an "A" are relevant to me.
The url I tried does look like this:

sftp://user:password@sftp_host/root/path/A*/*.zip

The outcome was as I expected, only files inside the dirs that start with an "A" were returned. But the performance was still bad so I decided to debug the whole thing and discovered that the filesystem still lists all subdirectories regardless of their name. I had a look into the implementation of the glob() function in the AbstractFileSystem and believe that these lines are responsible for that:

allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
# Escape characters special to python regex, leaving our supported
# special characters in place.
# See https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html
# for shell globbing details.
pattern = (
    "^"
    + (
        path.replace("\\", r"\\")
        .replace(".", r"\.")
        .replace("+", r"\+")
        .replace("//", "/")
        .replace("(", r"\(")
        .replace(")", r"\)")
        .replace("|", r"\|")
        .replace("^", r"\^")
        .replace("$", r"\$")
        .replace("{", r"\{")
        .replace("}", r"\}")
        .rstrip("/")
        .replace("?", ".")
    )
    + "$"
)
pattern = re.sub("/[*]{2}", "=SLASH_DOUBLE_STARS=", pattern)
pattern = re.sub("[*]{2}/?", "=DOUBLE_STARS=", pattern)
pattern = re.sub("[*]", "[^/]*", pattern)
pattern = re.sub("=SLASH_DOUBLE_STARS=", "(|/.*)", pattern)
pattern = re.sub("=DOUBLE_STARS=", ".*", pattern)
pattern = re.compile(pattern)

out = {
    p: allpaths[p]
    for p in sorted(allpaths)
    if pattern.match(p.replace("//", "/").rstrip("/"))
}

It seems the filesystem first looks up all files and filters them afterwards. While this certainly delivers the expected results it is still slow as the server has to list all directories nevertheless.

I would like to propose to change the implementation and make it more performant, not only for SFTP.
My first naive idea is to expand the existing find() and walk() functions (or add new ones) with the ability to apply regex filters while traversing the tree. Therefore the globbed path could be split up and a regex could be created for every subdirectory, as it is the case right now with the complete path. Then this list of regex strings could be used to filter the path of every subdirectory.

Please tell me what you think and if this could be a possible solution. In case I missed or overlooked something feel free to correct me!

Start by reading AbstractFileSystem.glob() and the related find() and walk() implementations. Trace how the SFTP example is traversed and filtered, then assess how per-directory glob or regex filtering could be introduced without changing expected results. Done means the proposed traversal avoids unnecessary directory listings and preserves existing glob behavior across supported filesystems.