Increase glob performance by utilizing prefixes

Currently doing this results in a very poor query, listing every object (I think just in some-folder, but i have not looked too close) and then filtering in python.

fs.glob("s3://bucket-name/some-folder/some-file-*

Would it be possible to automatically add everything before the first wildcard as a prefix? Equivalent to

fs.find("s3://bucket-name/some-folder", prefix="some-file-")

I could imagine a more complex optimization here for when people use multiple * and **, but a cheap fix would help in a large number of use cases.

An real world example of running into this would be looking at cloudfront logs. AWS stores these in the format <user-controlled-directory>/<cloudfront-distribution-id>.%Y-%m-%d-%h-<random-int>.gz

If I wanted to work on a single day of these using s3fs + dask, the standard way would be

dd.read_csv("s3://bucket-name/logs/distributionid.2022-01-04-*")

Due to the way s3fs seems to be handling this, it results in paginating an potentially extremely large number of files (millions). My hacky workaround was to use find and run read_csv on each file individually, stitching them together with concat. Even this less than optimal workaround resulted in saving many minutes just paginating list results.

Dask or pandas could probably handle this upstream but i'm not sure that's the right place for this kind of optimization?

Start from the fs.glob and fs.find entry points described in the issue, then trace how S3 listings are requested and filtered for wildcard paths. The change is done when a literal prefix before the first wildcard limits listing without changing glob results, including the shown CloudFront-log pattern; add or update coverage for these cases.