Support operations on byte arrays
- Dominant language
- Rust
- Stars
- 2.6k
- Forks
- 179
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 29
Description
## Progress
- [ ] Add `ByteArray` as private type
- [ ] Use `ByteArray` internally to experiment
- [ ] Add public uses of `ByteArray`
## Motivation
We want to be able to support operations on byte arrays, especially `[u8; size_of::()]` for some zerocopy-compatible type `T`. E.g.:
```rust
trait FromBytes {
fn from_bytes(bytes: [u8; size_of::()]) -> Self
where Self: Sized;
fn ref_from_bytes(bytes: &[u8; size_of::()]) -> Self
where Self: Sized + Unaligned + Immutable;
}
trait AsBytes {
fn into_bytes(self) -> [u8; size_of::()]
where Self: Sized;
fn as_bytes(&self) -> &[u8; size_of::()]
where Self: Sized + Immutable;
}
let t: T = transmute!([0u8; size_of::()]);
```
A lot of code both inside zerocopy and in user code currently has no way to reason about size equality on byte slices, and so ends up re-doing bounds checks. Consider this code from Fuchsia's [packet crate](https://fuchsia-docs.firebaseapp.com/rust/packet/index.html):
```rust
fn take_obj_front(&mut self) -> Option>
where
T: Unaligned,
{
let bytes = self.take_front(mem::size_of::())?;
// new_unaligned only returns None if there aren't enough bytes
Some(Ref::new_unaligned(bytes).unwrap())
}
```
In this code, `self.take_front` returns `Option<&[u8]>`, and so the type system can't reason about the returned byte slice satisfying `bytes.len() == size_of::()`.
As part of #1315, we'd like to support the general pattern of reading or writing objects to byte slices or fancier buffer types. Given support for byte arrays, we could write something like:
```rust
trait Buffer {
fn take_bytes_front(&mut self) -> Option<&[u8; N]>;
fn take_obj_front(&mut self) -> Option<&T> {
let bytes = self.take_bytes_front::<{size_of::()}>()?;
Some(transmute_ref!(bytes))
}
}
```
## Stabilize `size_of::()`
One approach we could take to accomplish this would be to stabilize `size_of::()` for use in a type in a generic context (a special case of [generic_const_exprs](https://github.com/rust-lang/rust/issues/76560).
## Polyfill
Another approach - that we can implement on our own without being blocked on Rust - is to add a polyfill type like the following
```rust
/// An array of `size_of::()` bytes.
///
/// Since the `generic_const_exprs` feature is unstable, it is not possible
/// to use the type `[u8; size_of::()]` in a context in which `T` is
/// generic. `ByteArray` fills this gap.
///
/// # Layout
///
/// `ByteArray` has the same layout and bit validity as `[u8; size_of::()]`.
#[derive(FromBytes, Unaligned)]
#[repr(transparent)]
pub struct ByteArray(
// INVARIANT: All of the bytes of this field are initialized.
Unalign>,
);
impl ByteArray {
// Not necessarily public. This is where we write the unsafe code that understands
// that `size_of::() == size_of::>()` since the type system itself
// isn't smart enough to understand that (at least when `T` is generic).
fn as_t(&self) -> Ptr<'_, T, (invariant::Shared, invariant::Any, invariant::Initialized)> {
let ptr = Ptr::from_ref(self);
// SAFETY: TODO
let ptr = unsafe { ptr.cast_unsized(|b| b as *mut T) };
// SAFETY: By safety invariant on `ByteArray`, `ByteArray` has the same bit validity
// as `[u8; _]`, which requires its bytes all be initialized.
unsafe { ptr.assume_initialized() }
}
}
```
Using this polyfill, we could write the `Buffer` trait from the motivation section as:
```rust
trait Buffer {
fn take_bytes_front(&mut self) -> Option<&ByteArray>;
fn take_obj_front(&mut self) -> Option<&T> {
let bytes = self.take_bytes_front::()?;
Some(bytes.as_t())
}
}
```
*If we use a type which supports unsized types (`Unalign` doesn't), we could even make this more powerful than `[u8; size_of::()]`. For `T: Sized`, `ByteArray` would have the same layout as `T`, but for `T: !Sized`, it would have a layout closer to `[u8]`. It's unclear how an unsized version of this could be constructed, though, since the fat pointer would need to know the number of trailing slice elements in `T`, not the number of bytes.*
### Interior mutability
TODO: Explain why Stacked Borrows would require `T: Immutable`, but why we may not need that bound in practice (ie, we can "disable" interior mutability).
d a `&ByteArray` to the same memory if `T` contained an `UnsafeCell`.*
This was originally prototyped (though never merged) [here](https://fuchsia-review.googlesource.com/c/fuchsia/+/653691).
TODO: Is it possible to support `T: ?Sized`? `MaybeUninit` requires `T: Sized`, and in general, unions don't support unsized types, so it's not possible to just manually implement a standin `MaybeUninit` that *does* support `T: ?Sized`.
Contributor guide
Assessment
This issue has not been assessed yet.