Memory Allocation?

This downloader loads each batch into memory before writing the files to disk, since our initial storage plan was to use `HDF5`.
Since we decided to transfer the dataset into either `HDF5` or `webdataset` format _**after**_ completing the download to avoid data backup complications and difficulties with parallel data writes, this setup is open to change.

The question is whether it is worth changing to writing image data as it comes down to reduce the memory bottleneck and utilize cores more efficiently.

The issue describes a downloader that loads each batch into memory before writing image files, but it names no source files, tests, or entry points. Start by locating the batch download and disk-write path, then evaluate whether streaming writes are compatible with the current parallel workflow. Done would require a decided approach and a verified reduction in the memory bottleneck.

Written by the indexing model from the issue text.