google / google/zerocopy

Support operations on byte arrays

Open
#248 0 comments 0 reactions 0 assignees View on GitHub
blocked-on-generic_const_exprs blocked-on-rust compatibility-nonbreaking
Dominant language
Rust
Stars
2.6k
Forks
179
Avg merge
1d 19h
Merged PRs (30d)
29

Description

## Progress

- [ ] Add `ByteArray` as private type
- [ ] Use `ByteArray` internally to experiment
- [ ] Add public uses of `ByteArray`

## Motivation

We want to be able to support operations on byte arrays, especially `[u8; size_of::()]` for some zerocopy-compatible type `T`. E.g.:

```rust
trait FromBytes {
fn from_bytes(bytes: [u8; size_of::()]) -> Self
where Self: Sized;
fn ref_from_bytes(bytes: &[u8; size_of::()]) -> Self
where Self: Sized + Unaligned + Immutable;
}

trait AsBytes {
fn into_bytes(self) -> [u8; size_of::()]
where Self: Sized;
fn as_bytes(&self) -> &[u8; size_of::()]
where Self: Sized + Immutable;
}

let t: T = transmute!([0u8; size_of::()]);
```

A lot of code both inside zerocopy and in user code currently has no way to reason about size equality on byte slices, and so ends up re-doing bounds checks. Consider this code from Fuchsia's [packet crate](https://fuchsia-docs.firebaseapp.com/rust/packet/index.html):

```rust
fn take_obj_front(&mut self) -> Option>
where
T: Unaligned,
{
let bytes = self.take_front(mem::size_of::())?;
// new_unaligned only returns None if there aren't enough bytes
Some(Ref::new_unaligned(bytes).unwrap())
}
```

In this code, `self.take_front` returns `Option<&[u8]>`, and so the type system can't reason about the returned byte slice satisfying `bytes.len() == size_of::()`.

As part of #1315, we'd like to support the general pattern of reading or writing objects to byte slices or fancier buffer types. Given support for byte arrays, we could write something like:

```rust
trait Buffer {
fn take_bytes_front(&mut self) -> Option<&[u8; N]>;

fn take_obj_front(&mut self) -> Option<&T> {
let bytes = self.take_bytes_front::<{size_of::()}>()?;
Some(transmute_ref!(bytes))
}
}
```

## Stabilize `size_of::()`

One approach we could take to accomplish this would be to stabilize `size_of::()` for use in a type in a generic context (a special case of [generic_const_exprs](https://github.com/rust-lang/rust/issues/76560).

## Polyfill

Another approach - that we can implement on our own without being blocked on Rust - is to add a polyfill type like the following

```rust
/// An array of `size_of::()` bytes.
///
/// Since the `generic_const_exprs` feature is unstable, it is not possible
/// to use the type `[u8; size_of::()]` in a context in which `T` is
/// generic. `ByteArray` fills this gap.
///
/// # Layout
///
/// `ByteArray` has the same layout and bit validity as `[u8; size_of::()]`.
#[derive(FromBytes, Unaligned)]
#[repr(transparent)]
pub struct ByteArray(
// INVARIANT: All of the bytes of this field are initialized.
Unalign>,
);

impl ByteArray {
// Not necessarily public. This is where we write the unsafe code that understands
// that `size_of::() == size_of::>()` since the type system itself
// isn't smart enough to understand that (at least when `T` is generic).
fn as_t(&self) -> Ptr<'_, T, (invariant::Shared, invariant::Any, invariant::Initialized)> {
let ptr = Ptr::from_ref(self);
// SAFETY: TODO
let ptr = unsafe { ptr.cast_unsized(|b| b as *mut T) };
// SAFETY: By safety invariant on `ByteArray`, `ByteArray` has the same bit validity
// as `[u8; _]`, which requires its bytes all be initialized.
unsafe { ptr.assume_initialized() }
}
}
```

Using this polyfill, we could write the `Buffer` trait from the motivation section as:

```rust
trait Buffer {
fn take_bytes_front(&mut self) -> Option<&ByteArray>;

fn take_obj_front(&mut self) -> Option<&T> {
let bytes = self.take_bytes_front::()?;
Some(bytes.as_t())
}
}
```

*If we use a type which supports unsized types (`Unalign` doesn't), we could even make this more powerful than `[u8; size_of::()]`. For `T: Sized`, `ByteArray` would have the same layout as `T`, but for `T: !Sized`, it would have a layout closer to `[u8]`. It's unclear how an unsized version of this could be constructed, though, since the fat pointer would need to know the number of trailing slice elements in `T`, not the number of bytes.*

### Interior mutability

TODO: Explain why Stacked Borrows would require `T: Immutable`, but why we may not need that bound in practice (ie, we can "disable" interior mutability).
d a `&ByteArray` to the same memory if `T` contained an `UnsafeCell`.*

This was originally prototyped (though never merged) [here](https://fuchsia-review.googlesource.com/c/fuchsia/+/653691).

TODO: Is it possible to support `T: ?Sized`? `MaybeUninit` requires `T: Sized`, and in general, unions don't support unsized types, so it's not possible to just manually implement a standin `MaybeUninit` that *does* support `T: ?Sized`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.