PyO3 / PyO3/pyo3

Consider making it possible to borrow `PyRef` in `#[pyclass]`

Open
#1,089 20 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

1.0-candidate hard needs-design
Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

At the moment, as #1088 notes (and also #1085), it's difficult to expose things like iterators to Python because of the restriction that #[pyclass] cannot take a lifetime.

This restriction obviously makes sense because Rust-managed data cannot be borrowed and then exposed to Python; Python will allow this data to live arbitrarily long.

However, data that's already owned by Python, e.g. in a PyCell<T>, can be safely "borrowed" and stored in another #[pyclass]. We do this already, but only for the lifetime-free case Py<T>.

It would be incredibly powerful if PyRef<'a, T> (or some similar construct) was able to be stored inside another #[pyclass]. This would make make wrapping arbitrary iterators, for example, much easier.

This playground shows that with some carefully-placed transmutes it's possible to hack the type system to at least achieve the desired affect: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=041f9a6f5dfa3a5fc59ece7105cc4dfd

That code is a wildly unsafe sketch that doesn't actually use the PyO3 types, so please don't try to use it as-is. You're responsible for all segfaults if you do 😄. If you're finding yourself in need of this feature, I can offer mentoring to help design how PyO3 could make such a trick safe and ergonomic.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the discussion in issues #1088 and #1085, then study the PyRef, PyCell, and #[pyclass] concepts described here. The linked Rust playground illustrates the lifetime challenge but is explicitly unsafe and not an implementation. Done means defining and documenting a safe, ergonomic way to store borrowed Python-owned data in another #[pyclass].

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.