batch downloads of mixed filesystems

Hello, I'm using fsspec with STAC metadata as a way to download assets in STAC Items in a standard way. These assets are most commonly http or s3 but could potentially be others.

A common scenario is objects in a requester pays bucket. In STAC though, we may have multiple assets we want to download where some are s3 in requester pays buckets and some aren't, or are http. See this Sentinel item as an example.

I'm also implementing this to support async downloads.

The problem is that I want to be able to download all assets from an item (or maybe even multiple items). So I currently loop through the assets, use url_to_fs to create the fs object and create an async task with that. But in the case of requester pays we need to specify that, but we can't pass requester_pays to url_to_fs when the URL ends up being https, as it throws an error.

The relevant code is here.

I'm looking for any suggestions on how to handle this without having to pre-determine the fs myself from the URL. url_to_fs is of great utility, but what I really want is for it to ignore keywords that aren't relevant for the resulting fs.

Thanks in advance for any thoughts!

Start with url_to_fs and the referenced stactask/asset_io.py around line 63 to understand how mixed HTTP and S3 assets are converted into filesystem objects. Done means an asset batch can include requester-pays S3 objects alongside other assets, including async downloads, without requiring the caller to preselect each filesystem or passing irrelevant options to another filesystem.