JuliaPy / JuliaPy/PyCall.jl

PyArray not converting a PyObject with __array_interface__ property, pointer, and writable?

Abierto
#800 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Julia
Estrellas
1.5k
Forks
186
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

I am trying to obtain a Julia ndarray from a Python interface to a C++ program ([Psi4](/http://www.psicode.org)). Psi4 exports its internal arrays to Python through NumPy Array interface, but one needs to use `np.asarray` for them to behave as normal NumPy arrays. I get the matrix with:

```julia
julia> matrix = psi4.core.Matrix(3,3)

julia> typeof(matrix) # I would expect some Julia Matrix type
PyObject
```
The returned object is not a Julia ndarray, as I expected. Since it is not a traditional NumPy array I did the transformation
with NumPy

```julia
julia> npmatrix = np.asarray(matrix) # delivers a Julia matrix, but ...
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0

julia> npmatrix[1,2] = 4
4

julia> matrix.np # ... npmatrix is a copy, not a view
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
```
In Python you get a view, but in Julia a copy. Looking inside I found

```julia
julia> matrix.__array_interface__ # It certainly has an array_interface
Dict{Any,Any} with 6 entries:
"shape" => (3, 3) # dimensions (C ordering)
"strides" => nothing # C ordering
"typestr" => " (140701706682928, false) # ptr to first element and writable!
"descr" => [("", " 3
```
It certainly has a NumPy array interface, and it is writable. So I thought that this could be transformed with `PyArray`.

```julia
julia> PyArray(matrix)
ERROR: PyError ($(Expr(:escape, :(ccall(#= .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:124 =# @pysym(:PyObject_GetBuffer), Cint, (PyPtr, Ref{PyBuffer}, Cint), o, b, flags)))))
BufferError('pybind11_getbuffer(): Internal error')

Stacktrace:
[1] pyerr_check at ../.julia/packages/PyCall/zqDXB/src/exception.jl:60 [inlined]
[2] pyerr_check at .../.julia/packages/PyCall/zqDXB/src/exception.jl:64 [inlined]
[3] _handle_error(::String) at .../.julia/packages/PyCall/zqDXB/src/exception.jl:81
[4] macro expansion at .../.julia/packages/PyCall/zqDXB/src/exception.jl:95 [inlined]
[5] PyBuffer! at .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:124 [inlined]
[6] PyBuffer(::PyObject, ::Int32) at .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:119
[7] PyArray_Info(::PyObject) at .../.julia/packages/PyCall/zqDXB/src/pyarray.jl:16
[8] PyArray(::PyObject) at .../.julia/packages/PyCall/zqDXB/src/pyarray.jl:124
[9] top-level scope at REPL[10]:1
```
It seems that PyCall treats it as a buffer when it is not `PyCall.isbuftype(matrix) == false`.
However, I can define a function that accesses the metadata of the NumPy array to
wrap the memory block with a Julia array, although it is very naive, not general, and
probably something better is already in PyCall, maybe `PyArray` can with some guidance?

```julia
julia>function psi4view(psi4matrix) # My ad-hoc solution that assumes Float64, ...
array_interface = psi4matrix.__array_interface__
array_interface["data"][2] == false || @warn "Not writable"
array_interface["strides"] == nothing || @warn "Different ordering than C"
array_interface["typestr"] == " jlmatrix = psi4view(matrix) # It works
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
```

Effectively, it works (only for "lvalues"), as I want:

```julia
julia> jlmatrix .= 1 # Changing the Julia matrix
3×3 Array{Float64,2}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0

julia> matrix.np # changes the original matrix. That is what I want.
3×3 Array{Float64,2}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0
```
I do not know what is impeding the transformation (maybe not being a normal NumPy array) but it is possible
and the data is there. I wonder if there is a way I can get the transformation using PyCall instead of relying on my solution.

For more context I have a Jupyter notebook [showing the scenario where it happened](https://github.com/zyth0s/psi4julia/blob/master/Tutorials/01_Psi4Julia-Basics/1c_psi4-numpy-datasharing.ipynb).
If you are interested in reproducing it, the [following instructions](https://github.com/zyth0s/psi4julia#getting-started) tell you the dependencies. I am using Julia 1.4.2 and PyCall v1.91.4.

Thank you in advance,
Daniel

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.