JuliaPy / JuliaPy/PyCall.jl

PyArray not converting a PyObject with __array_interface__ property, pointer, and writable?

オープン
#800 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Julia
スター
1.5k
フォーク
186
PR マージ指標
30日以内にマージされた PR はありません

説明

I am trying to obtain a Julia ndarray from a Python interface to a C++ program ([Psi4](/http://www.psicode.org)). Psi4 exports its internal arrays to Python through NumPy Array interface, but one needs to use `np.asarray` for them to behave as normal NumPy arrays. I get the matrix with:

```julia
julia> matrix = psi4.core.Matrix(3,3)

julia> typeof(matrix) # I would expect some Julia Matrix type
PyObject
```
The returned object is not a Julia ndarray, as I expected. Since it is not a traditional NumPy array I did the transformation
with NumPy

```julia
julia> npmatrix = np.asarray(matrix) # delivers a Julia matrix, but ...
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0

julia> npmatrix[1,2] = 4
4

julia> matrix.np # ... npmatrix is a copy, not a view
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
```
In Python you get a view, but in Julia a copy. Looking inside I found

```julia
julia> matrix.__array_interface__ # It certainly has an array_interface
Dict{Any,Any} with 6 entries:
"shape" => (3, 3) # dimensions (C ordering)
"strides" => nothing # C ordering
"typestr" => " (140701706682928, false) # ptr to first element and writable!
"descr" => [("", " 3
```
It certainly has a NumPy array interface, and it is writable. So I thought that this could be transformed with `PyArray`.

```julia
julia> PyArray(matrix)
ERROR: PyError ($(Expr(:escape, :(ccall(#= .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:124 =# @pysym(:PyObject_GetBuffer), Cint, (PyPtr, Ref{PyBuffer}, Cint), o, b, flags)))))
BufferError('pybind11_getbuffer(): Internal error')

Stacktrace:
[1] pyerr_check at ../.julia/packages/PyCall/zqDXB/src/exception.jl:60 [inlined]
[2] pyerr_check at .../.julia/packages/PyCall/zqDXB/src/exception.jl:64 [inlined]
[3] _handle_error(::String) at .../.julia/packages/PyCall/zqDXB/src/exception.jl:81
[4] macro expansion at .../.julia/packages/PyCall/zqDXB/src/exception.jl:95 [inlined]
[5] PyBuffer! at .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:124 [inlined]
[6] PyBuffer(::PyObject, ::Int32) at .../.julia/packages/PyCall/zqDXB/src/pybuffer.jl:119
[7] PyArray_Info(::PyObject) at .../.julia/packages/PyCall/zqDXB/src/pyarray.jl:16
[8] PyArray(::PyObject) at .../.julia/packages/PyCall/zqDXB/src/pyarray.jl:124
[9] top-level scope at REPL[10]:1
```
It seems that PyCall treats it as a buffer when it is not `PyCall.isbuftype(matrix) == false`.
However, I can define a function that accesses the metadata of the NumPy array to
wrap the memory block with a Julia array, although it is very naive, not general, and
probably something better is already in PyCall, maybe `PyArray` can with some guidance?

```julia
julia>function psi4view(psi4matrix) # My ad-hoc solution that assumes Float64, ...
array_interface = psi4matrix.__array_interface__
array_interface["data"][2] == false || @warn "Not writable"
array_interface["strides"] == nothing || @warn "Different ordering than C"
array_interface["typestr"] == " jlmatrix = psi4view(matrix) # It works
3×3 Array{Float64,2}:
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
```

Effectively, it works (only for "lvalues"), as I want:

```julia
julia> jlmatrix .= 1 # Changing the Julia matrix
3×3 Array{Float64,2}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0

julia> matrix.np # changes the original matrix. That is what I want.
3×3 Array{Float64,2}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0
```
I do not know what is impeding the transformation (maybe not being a normal NumPy array) but it is possible
and the data is there. I wonder if there is a way I can get the transformation using PyCall instead of relying on my solution.

For more context I have a Jupyter notebook [showing the scenario where it happened](https://github.com/zyth0s/psi4julia/blob/master/Tutorials/01_Psi4Julia-Basics/1c_psi4-numpy-datasharing.ipynb).
If you are interested in reproducing it, the [following instructions](https://github.com/zyth0s/psi4julia#getting-started) tell you the dependencies. I am using Julia 1.4.2 and PyCall v1.91.4.

Thank you in advance,
Daniel

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Start with pybuffer.jl:124 and pyarray.jl:16,124, then reproduce PyArray(matrix) using the Psi4 object and its __array_interface__. Determine why buffer conversion raises BufferError despite the interface metadata; done means PyArray supports this object as a writable view without copying, with coverage for the reported shape and type.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
numpy, python
領域
tooling
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。