Handle large ingests

There's an ERA5 dataset with over 500,000 files in it. This can't be ingested in the length of a standard batch system job. There are many ways to speed this:

* use parallelism to read and checksum multiple files at once
* allow ingestions to resume
* allow ingestions to be split into smaller chunks
* do all variables need to go into the same dataset?

No files, tests, or entry points are named. Start by locating the ingestion workflow and documenting how a standard batch job handles the 500,000-file ERA5 dataset; completion requires choosing and specifying a workable approach for parallelism, resuming, chunking, or variable separation.

Written by the indexing model from the issue text.