syncing subset of a directory using leaf file prefix is needlessly super slow

I have a bucket with a million files that are spread well in terms of their key pattern. (e.g., JF2SJAEC8HH466546.xml, 4S4BSANC7H3257612.xml, etc). Often I want to fetch a subset to my local disk that I would describe in s3cmd by "s3://mybucket/dir1/4S4B", using a prefix match. My understanding is that with s4cmd I instead pass "s4://mybucket/dir1/4S4B*" to the sync command, which is all fine to use the wildcard.

The problem is that it takes forever for s4cmd to figure out the set of files to copy. Running with --debug, it looks to me like it's enumerating the whole dir and matching my pattern on the client. Since I only want 100 out of a million files, this is a poorly performing approach for my case (that is, if my assumptions are right, I haven't read the code).

I would suggest the optimization where the source path up to the first wildcard should be used a prefix to the underlying AWS API, and the number of file names fetched would be be cut enormously. Although s3cmd was a pig copying each file one by one, I believe it was smarter about this first enumeration step.

Also, congrats on a good looking tool, I have been wanting something with threading on the cmdline for a long time.

Start with the sync command's source-path handling and the AWS object-listing call, then reproduce the wildcard case with a prefix such as `s3://mybucket/dir1/4S4B*`. Done means the portion before the first wildcard is passed as the listing prefix, so matching a small subset does not enumerate the entire directory.

Written by the indexing model from the issue text.