Why are find/glob functions not lazy?

Hello,

For example, I have a directory with subdirectories and a lot of files (tens/hundreds of millions total, tens of thousands in each directory). I want to read/process these files in parallel long hours and days. I want to select files by glob and pass paths to the workers through a queue for example. And I cannot use fsspec glob for this, because it is not a lazy function. Yes, I know about fsspec walk but I would use glob recursive pattern, something like this:

for path in fs.glob('/path/to/root/**.xml')
    queue.put(path)

I can make it with pathlib.Path.rglob for local, because it is a lazy function and it does not return a huge list of million files. However I cannot make it with an abstract filesystem via fsspec glob.

Is there a way to make these functions lazy or it is hard/impossible within the current design?

Start with the fsspec glob and walk entry points, then compare their behavior with pathlib.Path.rglob for the recursive pattern described. Determine whether a lazy glob interface fits the current filesystem design; done means callers can consume matching paths incrementally without materializing the full result.