openPMD / openPMD/openPMD-api

Using preallocated buffers in python

Open
#1,466 4 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

**Problem**
Hi! I was often running into the situation where I wanted to do sth like that:
```python
a=np.zeros((2, 2, 1728 , 100), dtype=np.float32)
a = np.ascontiguousarray(a)
a[0, :, :, :] = some_mrc[240:242, :, 100:200]
a[1, :, :, :] = other_mrc[240:242, :, 100:200]
series.flush()
```
So, reading different chunks to different parts of a predefined `numpy` array.
But this fails with: `ValueError: vector::_M_default_append`

There seams to exists an, as far as I know, undocumented workaround, that I just found:

```python
a=np.zeros((2, 2, 1728 , 100), dtype=np.float32)
a = np.ascontiguousarray(a)
some_mrc.load_chunk(a[0, :, :, :], offset=[240,0,100], extent=[2,1728,100])
other_mrc.load_chunk(a[1, :, :, :], offset=[240,0,100], extent=[2,1728,100])
series.flush()
```
As the `MeshRecordComponent.load_chunk` method can take something providing the python buffer interface as an extra argument. But this is not that easy to use as with the slicing syntax and the `[ ]` operator.

**Possible solution**

It would be fantastic if sth like this `a[0, :, :, :] = example_mrc[240:242, :, 100:200]` could work, but I'm not sure how one would change the function execution based on the object on the left-hand side (though `numpy` broadcasting seams to do it somehow).
Alternatively, one could have a new `MeshRecordComponent` method returning an object with a `__get_item__` method, so that we can use syntax like this:
```python
a=np.zeros((2, 2, 1728 , 100), dtype=np.float32)
a = np.ascontiguousarray(a)
some_mrc.get_buffer_loader(buffer=a[0, :, :, :])[240:242, :, 100:200]
other_mrc.get_buffer_loader(buffer=a[1, :, :, :])[240:242, :, 100:200]
series.flush()
```

Or maybe, there exists some class that is exposed in python that can be used to convert the slicing syntax to offset and extent? Sth like: `some_mrc.load_chunk(a[0, :, :, :], *io.offset_and_extent_getter[240:242, :, :100:200])`

I don't know much about how the python bindings work, otherwise I would implement it myself.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.