Grabbing whole dataset from s3 currently slow (possibly other cloud blob storage)

We use metaflow for orchestration of many of our ML tasks, and they have something that if you specify a path in s3 will grab the data extremely fast as it's done in parallel. It can't do a lot of the things like lance that allow filtering or selection of specific columns, but I think the download method on lance backend could change dependant on what you are trying to do? This is the code normally used for metaflow and I think if a filtering clause isn't specified you could implement something similar:

with S3(s3root=f"{source_s3_base_path}") as s3:
        files = s3.get_all()
        return pd.concat([pd.read_parquet(file.path) for file in files]) 

Just pointing it out because if you are outputting daily new fragments to append to the data. It's quite slow grabbing the whole dataset, where effectively the same operation using the code above is 20x~30x faster. This would get tricky with filtering or even specifying columns I think. Anyways, I can further point to the above code but just a possible future optimization especially if someone wants to pull down the whole dataset from blob storage. A caveat is this does download all those chunks in parallel to disk and then load them in, so you would need ample disk space.

No implementation file, entry point, or test is named. Start by locating the Lance backend path used to load an entire S3 dataset and compare it with the Metaflow S3 get_all example. Done requires an agreed scope for unfiltered reads versus filtering or column selection, plus tests demonstrating the intended behavior and performance.