Generate `numpy.ndarray` from iterable with automatic memory pre-allocation

Colab

I think a general use case of numpy.ndarrays is loading data from several files into one array, these could be pickles, images, or anything that can easily be loaded as a numpy.ndarray. In this case all these files would have the same shape and dtype, and would have normal strides. Is there an elegant way to load such files, avoiding duplicate code and memory overhead?

E.g. files created like this:

import numpy
from skimage.io import imsave, imread
from glob import glob

sz = (128, 128, 1)

for i, color in enumerate([[0, 0, 1], [0, 1, 0], [0, 1, 1], [1, 0, 0], [1, 0, 1], [1, 1, 0]]):
  nice_image = numpy.clip(
      (0.95 + 0.1 * numpy.random.randn(*sz)) * [[color]],
      0, 1)
  imsave(
    f'nice-image-{i}.png',
    (255 * nice_image).astype(numpy.uint8)
  )

One way to open the files would be like this, but it would need twice the memory that is actually needed to first store the list and then create the array.

images = numpy.concatenate([
  imread(filename)[numpy.newaxis] for filename in filenames
], axis=0)

Alternatively:

first_image = imread(filenames[0])

images = numpy.zeros((len(filenames), ) + im0.shape, dtype=im0.dtype)
images[0] = first_image

for image, filename in zip(images[1:], filenames[1:]):
  image[...] = imread(filename)

But this has two places where the data is loaded, so duplicate code, which makes it non-transparent and easily introduces bugs.

Would it make sense for the numpy api to have a fromiter_nd for example:

images = numpy.fromiter_nd((
    imread(filename)[numpy.newaxis]
    for filename in filenames
), axis=0, length=len(filenames))

Existing functions fail in the following manor:

• numpy.fromiter Assumes numbers and creates something 1D,
• numpy.stack, numpy.concatenate, do not inspect the first item and use probably known generator length information to preallocate memory, and hence need twice the memory at peak.

Start with the issue's examples and compare the behavior of numpy.fromiter, numpy.stack, and numpy.concatenate, then reproduce the generator and image-loading case from the linked Colab notebook. Done would require a scoped decision on whether and how NumPy should expose shape-aware preallocation, including the behavior for the proposed inputs.