Improving ergonomics of getting slices from a PyBuffer
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.2k
- Forks
- 1k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 66
Description
I'm writing a function that accepts anything that can create a read-only buffer:
fn validate_cbor(&self, py: Python<'_>, cbor: &PyAny) -> PyResult<()> {
let buffer = PyBuffer::<u8>::get(cbor)?;
if !buffer.readonly() {
return Err(PyValueError::new_err("Must be read-only byte buffer"));
}
let slice = buffer
.as_slice(py)
.ok_or_else(|| PyTypeError::new_err("Must be a contiguous sequence of bytes"))?;
// Safety: The slice is &[ReadOnlyCell<u8>]. A ReadOnlyCell has the same
// memory representation as the underlying data; it's
// #[repr(transparent)] newtype around UnsafeCell. And per Rust docs
// "UnsafeCell<T> has the same in-memory representation as its inner
// type T". So the main issue is whether the data is _really_ read-only.
// We do the read-only check above, and yes a caller can probably somehow
// lie, but if they do that, that's really their fault.
let cbor: &[u8] = unsafe { std::mem::transmute(slice) };
// ... custom code here
}
It's not clear to me why PyBytes::as_bytes() shouldn't return &[ReadOnlyCell<u8>] too. After all, there is nothing preventing a particularly malicious user from overwriting the data in a Python bytes object, just need to write a little bit of C. We need to trust callers to some extent in a world where they can write C.
So my feeling is that as_slice() can just return Some(&[T]), unless there's some reason to expect buffer API implementations to lie about their readonly flag.
As a follow-up, if that makes sense then maybe the pattern above could be simplified and built-in to PyO3.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the PyBuffer::as_slice() and PyBytes::as_bytes() APIs and the readonly buffer behavior described in the issue. Determine whether returning a slice is sound when callers can provide buffer implementations, and assess whether the validate_cbor pattern should become a built-in PyO3 API. Done means the safety contract and resulting ergonomics are clearly decided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100