JuliaPy / JuliaPy/PyCall.jl

handling arrays with missing values (with possible solution)

Open
#616 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
1.5k
Forks
186
PR merge metrics
No merged PRs in 30d

Description

I would like to plot (with PyPlot) an array containing missing values:

```julia
using PyPlot, Missings
d = [missing 1; 2 3]
pcolor(d)
```
Unfortunately, I get the following error with PyCall v1.18.5, PyPlot v2.6.3 and Julia v1.0.1:

```
PyError ($(Expr(:escape, :(ccall(#= /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw)))))
TypeError('unorderable types: PyCall.jlwrap() <= int()',)
File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 2761, in pcolor
**kwargs)
File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/__init__.py", line 1810, in inner
return func(ax, *args, **kwargs)
File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 5785, in pcolor
collection.autoscale_None()
File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/cm.py", line 375, in autoscale_None
self.norm.autoscale_None(self._A)
File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/colors.py", line 988, in autoscale_None
self.vmin = A.min()
File "/home/abarth/.local/lib/python3.5/site-packages/numpy/core/_methods.py", line 32, in _amin
return umr_minimum(a, axis, None, out, keepdims, initial)

Stacktrace:
[1] pyerr_check at /home/abarth/.julia/packages/PyCall/0jMpb/src/exception.jl:60 [inlined]
[2] pyerr_check at /home/abarth/.julia/packages/PyCall/0jMpb/src/exception.jl:64 [inlined]
[3] macro expansion at /home/abarth/.julia/packages/PyCall/0jMpb/src/exception.jl:84 [inlined]
[4] __pycall!(::PyCall.PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyCall.PyObject, ::Ptr{Nothing}) at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44
[5] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{Array{Union{Missing, Int64},2}}, ::Int64, ::Ptr{Nothing}) at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:22
[6] #pycall#88 at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:11 [inlined]
[7] pycall at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:86 [inlined]
[8] #pcolor#81(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Array{Union{Missing, Int64},2}) at /home/abarth/.julia/packages/PyPlot/fZuOQ/src/PyPlot.jl:179
[9] pcolor(::Array{Union{Missing, Int64},2}) at /home/abarth/.julia/packages/PyPlot/fZuOQ/src/PyPlot.jl:176
[10] top-level scope at In[1]:3
```

However, if declare the following convertion rule in PyCall, the plotting works:

```julia
using PyCall
using PyCall: PyObject
function PyObject(a::Array{Union{T,Missing},N}) where {T,N}
numpy_ma = pyimport("numpy")["ma"]
pycall(numpy_ma["array"], Any, coalesce.(a,zero(T)), mask=ismissing.(a))
end
pcolor(d)
```
The function `PyCall` creates essentially a numpy masked array. All missing values are replaced by zero and a mask is given to indicate which values are missing.

Can this be added to PyCall?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with PyCall's PyObject conversion entry point and the conversion call shown in the issue, then reproduce the PyPlot pcolor example using an array with missing values. Compare the proposed numpy.ma conversion with the behavior around pyfncall.jl:44 and PyPlot.jl:176-179; done means the example passes missing values as a masked NumPy array.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia, numpy, python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.