casey / casey/x-serialization-format
Up-front vs incremental validation
- Dominant language
- Rust
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I've envisaged the format as being "zero parse", that is to say that up-front parsing of an entire message is not necessary in order to use it. Accessing any field can be performed by loading and following relative offsets, which is both inexpensive and only required for fields that you actually access.
However, in order to make this safe with untrusted messages, these offsets _would_ need to be validated ahead of time. Essentially, instead of an expensive parsing step, an inexpensive validation step would be required up-front.
The reason for this is that I intended the API to expose references to objects in buffers. More concretely, if you have a top-level table `Foo`, the `Deserialize` method on a buffer would give you a reference pointing into the buffer's memory, a `&Foo`.
Let's say that `Foo` contains a field `bar` of type `Bar`. I wanted to provide a function with signature `Foo::bar(&self) -> `&Bar` to access the field.
This access would entail following an offset. However, since `Foo::bar` only has access to a `&Foo`, and _not_ the original buffer containing the message, `Foo::bar` would not be able to ensure that it is safe to follow the relative offset to `bar`.
This is why I have thought that an up-front validation step would be required. This up-front validation step would check that all offsets in the message (along with lengths, for variable sized values like strings and slices) did not point outside of the message, and thus could be accessed safely.
Validation would be cheap, amounting only to loading offsets and adding them to pointers, and checking that the result is in-bound. However, on very large messages containing a deep hierarchy of objects, this could get expensive. Additionally, if a message is mmapped into memory, this traversal might cause large amounts of the message to be paged in, which is undesirable.
There are two alternative APIs that I can imagine, both of which are less appealing, but which avoid requiring up-front validation:
1. The first requires always passing in the original buffer whenever you want to follow an offset. So, instead of `Foo::bar(&self) -> &Bar`, we would have `Foo::bar(&self, message: &[u8]) -> &Bar`. This would allow `Foo::bar` to only follow the offset in bar if it contained in `message`. Thus, validation would be moved to individual accesses instead of being required up-front. The downside is that this is a much less ergonomic API, and that any individual access might fail, as opposed to knowing, after validation succeeds, that all accesses will succeed.
2. The second requires eschewing references for handle objects. Instead of receiving a `&'message Foo`, when deserializing the message, the user would receive a struct `Foo<'message>`, whose contents would be:
```rust
struct Foo<'message> {
inner: &'message ActualFoo, // this could actually be `offset_to_foo: usize`
message: &'message [u8],
}
```
Then, `Foo::bar` could use `self.message` to ensure that the offset to `bar` is in-bounds.
Actually, writing this I've convinced myself that making every access potentially fallible is not desirable. Users will probably prefer to pay the cost of validation up front, and not have to worry about either handling errors or panicking on every access.
However, I think I'll leave this issue open, in case anyone feels differently.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.