JuliaPy / JuliaPy/PyCall.jl

pyjlwrap support for Pickle Serialization

Abierto
#863 5 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Julia
Estrellas
1.5k
Forks
186
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

PyCall works nicely for many use cases between Python and Julia. In particular, there is one that could be improved and very important for Data Scientist community. For example, I tried to use it for PySpark library and works very well for the basic use case. But, if the user needs to create a UDF (User Defined Functions), the user will have trouble to serialize the functions.
The UDFs, in this case, would help to many DSs reuse Julia code and call spark to do the heavy work. Have this enabled, would improve the usage of Julia in different scenarios.

To solve the current issues with UDF, PyObject needs to be serializable with Pickle. I don't have much idea how to solve this, but I have a simple use case that if we fix would improve towards this functionality:

**Example:**
```
using PyCall
pickle = pyimport("pickle")
pickle.dumps(x -> x + 1)
```

**Error:**
```
ERROR: PyError ($(Expr(:escape, :(ccall(#= /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw)))))
TypeError("cannot pickle 'PyCall.jlwrap' object")

Stacktrace:
[1] pyerr_check at /root/.julia/packages/PyCall/zqDXB/src/exception.jl:60 [inlined]
[2] pyerr_check at /root/.julia/packages/PyCall/zqDXB/src/exception.jl:64 [inlined]
[3] _handle_error(::String) at /root/.julia/packages/PyCall/zqDXB/src/exception.jl:81
[4] macro expansion at /root/.julia/packages/PyCall/zqDXB/src/exception.jl:95 [inlined]
[5] #110 at /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:43 [inlined]
[6] disable_sigint at ./c.jl:446 [inlined]
[7] __pycall! at /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:42 [inlined]
[8] _pycall!(::PyObject, ::PyObject, ::Tuple{var"#3#4"}, ::Int64, ::Ptr{Nothing}) at /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:29
[9] _pycall!(::PyObject, ::PyObject, ::Tuple{var"#3#4"}, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:11
[10] (::PyObject)(::Function; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:86
[11] (::PyObject)(::Function) at /root/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:86
[12] top-level scope at REPL[13]:1
```

Reference to UDF in Python: https://docs.databricks.com/spark/latest/spark-sql/udf-python.html

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Reproduce the pickle.dumps example from the issue and start by reading the PyObject handling and the pyfncall.jl path shown in the traceback. Done means PyObject-backed functions can be serialized for the described UDF use case, with the reported failure covered by a regression test.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
julia, python, spark
Área
backend, data-engineering
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
30/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.