PyO3 / PyO3/pyo3

Improving ergonomics of getting slices from a PyBuffer

Open
#2,824 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

I'm writing a function that accepts anything that can create a read-only buffer:

fn validate_cbor(&self, py: Python<'_>, cbor: &PyAny) -> PyResult<()> {
        let buffer = PyBuffer::<u8>::get(cbor)?;
        if !buffer.readonly() {
            return Err(PyValueError::new_err("Must be read-only byte buffer"));
        }
        let slice = buffer
            .as_slice(py)
            .ok_or_else(|| PyTypeError::new_err("Must be a contiguous sequence of bytes"))?;

        // Safety: The slice is &[ReadOnlyCell<u8>]. A ReadOnlyCell has the same
        // memory representation as the underlying data; it's
        // #[repr(transparent)] newtype around UnsafeCell. And per Rust docs
        // "UnsafeCell<T> has the same in-memory representation as its inner
        // type T". So the main issue is whether the data is _really_ read-only.
        // We do the read-only check above, and yes a caller can probably somehow
        // lie, but if they do that, that's really their fault.
        let cbor: &[u8] = unsafe { std::mem::transmute(slice) };
        
        // ... custom code here
}

It's not clear to me why PyBytes::as_bytes() shouldn't return &[ReadOnlyCell<u8>] too. After all, there is nothing preventing a particularly malicious user from overwriting the data in a Python bytes object, just need to write a little bit of C. We need to trust callers to some extent in a world where they can write C.

So my feeling is that as_slice() can just return Some(&[T]), unless there's some reason to expect buffer API implementations to lie about their readonly flag.

As a follow-up, if that makes sense then maybe the pattern above could be simplified and built-in to PyO3.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the PyBuffer::as_slice() and PyBytes::as_bytes() APIs and the readonly buffer behavior described in the issue. Determine whether returning a slice is sound when callers can provide buffer implementations, and assess whether the validate_cbor pattern should become a built-in PyO3 API. Done means the safety contract and resulting ergonomics are clearly decided.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.