Support wrapping a Java-side block of memory allocated off-heap to Python
- Dominant language
- Python
- Stars
- 534
- Forks
- 95
- PR merge metrics
- No merged PRs in 30d
Description
With `ByteBuffer.allocateDirect` you can allocate memory off-heap in Java, which can then be shared with other processes. We want to easy manufacturing of numpy arrays and xarrays that wrap this sort of off-heap memory, so that you can directly change data in Python that originated in Java (by some definition of "originated"—since it's off-heap).
In #73, @hanslovsky wrote:
> should be possible already, albeit I don't think there is a convenience method for that. Things may have changed since I was last involved with imglyb (it was still pyjnius back then), but there is/was a way to generate ImgLib2 ArrayImgs backed by native memory and you can then simply pass that pointer into a numpy array.
See also https://github.com/imglib/imglib2-cache-python
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the ByteBuffer.allocateDirect use case in this issue, then read issue #73 and the linked imglib2-cache-python project for existing native-memory and pointer-handling approaches. The payload names no implementation files or tests; done would mean a documented, tested way to create NumPy or xarray views over Java off-heap memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, numpy, python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100