Large arrays of data

I have been looking for ways to deal with large amounts of data. As far as I understand, at the moment an array is initialized that is made to store all of the data that is generated from a loop. However, for very large arrays of data, this is not a viable approach. For instance, during an overnight measurement, if I want to save all the individual traces, this could amount to over a billion datapoints, which means several GB of memory. However, this data does not need to be stored in memory, and can in fact be forgotten as soon as it is stored on the local disk.

I was therefore thinking of alternate solutions to this problem, and was hoping people could help me with ideas. I think one possible approach would be to have the data_manager somehow "know" when to use the standard approach (initialize a full array for each gettable element), and when to use a different approach, such as having buffers, like an ATS digitizer. The buffers are continuously filled, and simultaneously stored. Once a buffer has been stored, it can be safely overwritten.

As to how the data manager should "know" about what approach to choose, my preference would go out to manually having to specify it, although some sort of automated choice depending on the array size would also be possible. In any case, I think a good additional check would be to verify that the data arrays do not exceed the memory-imposed limit.

I would like to know other people's opinions on the matter, and suggestions as to how this should be tackled. How could this best be integrated with the current QCoDeS framwork, and do people know any pre-existing packages, or some tricks, to deal with this efficiently and effectively? Furthermore, is any effort spent on this issue going to be useful, or is it going to be wasted effort because it probably won't be compatible anyway with the upcoming revamping of multiprocessing/multithreading (@giulioungaretti)?

Start by reviewing the current data_manager behavior and how it initializes arrays for loop-generated data. Consider the proposed buffers, local-disk storage, memory-limit checks, and compatibility with the planned multiprocessing/multithreading revamp; done requires an agreed integration design and implementation scope.

Written by the indexing model from the issue text.