'Open' stream example ?

The package contains examples of creating ArrayStream objects given a schema and a vector or list of arrays. That helps for chunks returned via, say, RecordBatchReader as this may not require the contiguous memory an unchunked approach would need. But as we instantiate with the whole vector (or list) we still require a similar total amount of memory at instantiation.

But can we create, say, a RecordBatchReader is a more 'streaming' fashion? Could be hand this back to the caller with only the initially-known list of Arrays _and also support further data_ ? So say the first call of `next()` would be covered but thereafter a more 'lazy' approach is used and RecordBatchReader supplies updates in true batches. Obviously a more complicated setup, but is something like this feasible / supported / planned / ... ?

I may be explaining myself poorly here but are there other references in the Arrow context that handle this is as a more 'open' subscription (in the sense of 'total payload unknown at instantiation') with a later callbacks to provide chunked updates? Or do I have the wrong mental model and should rather think about, say, a pub/sub model where a 'middle man' holds on to the data and passes is along? (I have done such things with Redis.)

Thanks in advance for any pointers, and apologies for posting such a vague and rambling issue.

Start with the existing ArrayStream examples and the RecordBatchReader references in the package. Review how the Arrow C Data and C Stream interfaces handle data whose total size is unknown, then determine whether a lazy or callback-driven reader is supported or would require a new design. Done means documenting the supported approach or the scope of a proposed one.

Written by the indexing model from the issue text.