JuliaPy / JuliaPy/PyCall.jl

Support Py_LIMITED_API in separate package as a dependency to PyCall.jl

Open
#714 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
1.5k
Forks
186
PR merge metrics
No merged PRs in 30d

Description

The Python C ABI/API has stability guarantees for a limited number of structs and functions.

https://www.python.org/dev/peps/pep-0384/

It'd be nice to support this as a separate package and use that package as a dependency in PyCall.jl. This separate package can implement the Py_LIMITED_API structs and functions without invoking the Python interpreter.

A (very early) prototype is located here:

https://github.com/kdheepak/Python.jl/blob/33b98177b0e624588a3f46ccfa96b317d57f1201/src/Python.jl

The motivation for this is so that we can build a shared object from a Julia file using [PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl) that is also a Python module. See screenshot below as an [example](https://github.com/JuliaLang/PackageCompiler.jl/blob/f40fb987c7e5f3d6c6ad9245f957af90b78bf575/hello.jl):

Screen Shot 2019-09-24 at 11 02 34 PM

No linking to the Python library occurs at compile time and the Python module does not embed a Python interpreter; it is a pure Julia shared library. This allows one to create a Python module without having Python installed and the structs and functions described in the Py_LIMITED_API would be sufficient to make this work.

Taking this a step further, we could allow for something like the following in the future (See https://github.com/yglukhov/nimpy and https://github.com/PyO3/PyO3 for similar projects in Nim and Rust):

**Write Julia code**

```julia
# hello.jl
using Python

@pycallable function hello(name::String)::Cvoid
println("Hello $name!")
end
```

**Compile**

```bash
$ julia --project juliac.jl -vasij `pwd`/hello.jl > /dev/null
```

**Run in Python**

```bash
$ ipython
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import hello

In [2]: hello.hello("John")
Hello John!

In [3]:
```

My understanding is that this would pose some challenges with the current setup in PyCall.jl since PyCall.jl initializes a interpreter in [`__init__`](https://github.com/JuliaPy/PyCall.jl/blob/9f4387535936f275e3582ebdc87cc2ef1f210030/src/pyinit.jl#L128) and when PackageCompiler runs, the references to that interpreter are embedded in the shared library.

Comments?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading PyCall.jl's src/pyinit.jl at __init__, then compare its initialization behavior with the early src/Python.jl prototype and the PEP 384 limited API. Done would require a defined separate dependency and a demonstrated PackageCompiler workflow that produces a Python module without embedding or linking a Python interpreter.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.