JuliaPy / JuliaPy/PyCall.jl

PyArray not converting a PyObject with __array_interface__ property, pointer, and writable?

Offen
#800 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Julia
Sterne
1.5k
Forks
186
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

I am trying to obtain a Julia ndarray from a Python interface to a C++ program ([Psi4](/http://www.psicode.org)). Psi4 exports its internal arrays to Python through NumPy Array interface, but one needs to use `np.asarray` for them to behave as normal NumPy arrays. I get the matrix with:

```julia
julia> matrix = psi4.core.Matrix(3,3)

julia> typeof(matrix) # I would expect some Julia Matrix type
PyObject
```
The returned object is not a Julia ndarray, as I expected. Since it is not a traditional NumPy array I did the transformation
with NumPy

```julia
julia> npmatrix = np.asarray(matrix) # delivers a Julia matrix, but ...
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0

julia> npmatrix[1,2] = 4
4

julia> matrix.np # ... npmatrix is a copy, not a view
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
```
In Python you get a view, but in Julia a copy. Looking inside I found

```julia
julia> matrix.__array_interface__ # It certainly has an array_interface
Dict{Any,Any} with 6 entries:
"shape" => (3, 3) # dimensions (C ordering)
"strides" => nothing # C ordering
"typestr" => " (140701706682928, false) # ptr to first element and writable!
"descr" => [("", " 3
```
It certainly has a NumPy array interface, and it is writable. So I thought that this could be transformed with `PyArray`.

```julia
julia> PyArray(matrix)
ERROR: PyError ($(Expr(:escape, :(ccall(#= .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:124 =# @pysym(:PyObject_GetBuffer), Cint, (PyPtr, Ref{PyBuffer}, Cint), o, b, flags)))))
BufferError('pybind11_getbuffer(): Internal error')

Stacktrace:
[1] pyerr_check at ../.julia/packages/PyCall/zqDXB/src/exception.jl:60 [inlined]
[2] pyerr_check at .../.julia/packages/PyCall/zqDXB/src/exception.jl:64 [inlined]
[3] _handle_error(::String) at .../.julia/packages/PyCall/zqDXB/src/exception.jl:81
[4] macro expansion at .../.julia/packages/PyCall/zqDXB/src/exception.jl:95 [inlined]
[5] PyBuffer! at .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:124 [inlined]
[6] PyBuffer(::PyObject, ::Int32) at .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:119
[7] PyArray_Info(::PyObject) at .../.julia/packages/PyCall/zqDXB/src/pyarray.jl:16
[8] PyArray(::PyObject) at .../.julia/packages/PyCall/zqDXB/src/pyarray.jl:124
[9] top-level scope at REPL[10]:1
```
It seems that PyCall treats it as a buffer when it is not `PyCall.isbuftype(matrix) == false`.
However, I can define a function that accesses the metadata of the NumPy array to
wrap the memory block with a Julia array, although it is very naive, not general, and
probably something better is already in PyCall, maybe `PyArray` can with some guidance?

```julia
julia>function psi4view(psi4matrix) # My ad-hoc solution that assumes Float64, ...
array_interface = psi4matrix.__array_interface__
array_interface["data"][2] == false || @warn "Not writable"
array_interface["strides"] == nothing || @warn "Different ordering than C"
array_interface["typestr"] == " jlmatrix = psi4view(matrix) # It works
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
```

Effectively, it works (only for "lvalues"), as I want:

```julia
julia> jlmatrix .= 1 # Changing the Julia matrix
3×3 Array{Float64,2}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0

julia> matrix.np # changes the original matrix. That is what I want.
3×3 Array{Float64,2}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0
```
I do not know what is impeding the transformation (maybe not being a normal NumPy array) but it is possible
and the data is there. I wonder if there is a way I can get the transformation using PyCall instead of relying on my solution.

For more context I have a Jupyter notebook [showing the scenario where it happened](https://github.com/zyth0s/psi4julia/blob/master/Tutorials/01_Psi4Julia-Basics/1c_psi4-numpy-datasharing.ipynb).
If you are interested in reproducing it, the [following instructions](https://github.com/zyth0s/psi4julia#getting-started) tell you the dependencies. I am using Julia 1.4.2 and PyCall v1.91.4.

Thank you in advance,
Daniel

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.