Structs vs Tables

I've been thinking about the difference between structs and tables, and I've started to think that perhaps `data` should add enough features to structs such that tables don't need special syntax.

These are the important features of tables that I can think of:

_Backwards compatibility_: If a field is added in a new version of a client, and that client serializes a table with that field, old clients will ignore it.

_Forwards compatibility_: If a field is added in a new version of a client, and that client deserializes a table without that field, it will get a `None` value instead of something bad happening.

_Deprecation_: Fields can be marked as deprecated, so that they won't be loaded/deserialized if present. Additionally, if all versions of all clients mark a field as deprecated, that field can be reused in a future client.

_Indirection_: Tables members are stored with some kind of indirection, so when client code gets a table value, there hasn't been any kind of traversal/deserialization of its members, until a getter for that member is called. (In contrast with structs, which are loaded into a language-native struct, which implies that all fields must be themselves loaded.)

_Compactness_: Tables are sometimes represented as arrays of offsets (FIDL) or vtables (flatbuffers), which makes a table with a bunch of optional values much more compact than the equivalent struct.

_Deffered validation_: Table fields are often a accessed with a getter. This could be used to defer validation of fields, as opposed to a struct, since structs are exposed to client code as native structs, so all fields must be validated and loaded.

_Default values_: Since table values may be missing, table fields may have default values, which are returned when deserializing a table missing that field.

_Repurposing fields_: Clients can mark a table field as deprecated, and then when there are no other clients in the wild using that field, new clients can re allocate that field with a new type.

If I'm missing any, do let me know!

Structs could be extended to support these with the addition of "flexible structs":

The fields of a struct can be annotated with `#[index(N)]`, where `N` is a non-negative integer. If any field has this annotation, all fields must have this annotation. `N` may not be repeated, and there may not be any gaps, although they may appear in any order in the source code. This changes a struct to a `flexible struct`. Flexible structs are encoded in a buffer with an additional integer, which is 1 + the largest index on the struct when it was encoded, this is called the "field count".

_Backwards compatibility, forwards compatibility, and default fields_: A new field may be added to a struct, but the type must be `Option`, or a default value must be provided with `#[default(VALUE)]`. When client code deserializes a value with a field count that is less than the number of fields that it knows about, missing fields are filled in with `None` or the default value given. If a is field missing that has no default, or is not `Option`, then it gives up and returns a deserialization error.

_Deprecation_: The type of a field may be wrapped with `data::Ignore`, then the deserializer will correctly account for space taken up by that field in structs that it is deserializing, but will not actually touch the value or do anything with it. `data::Ignore` fields must have a default value when serializing, so the encoder is able to write a valid value to that encoded field.

_Indirection_: Any type can be wrapped with `data::Ref`, this changes the representation of the type in a loaded struct (not in an encoded struct!) to `struct Ref<'a, T> { buffer: &'a [u8], offset: usize }`. When loading a struct into memory that has a `Ref` field, the field will always be the size of a slice and offset, regardless of the size of that field. This does not change the encoded representation, so can be added and removed without breaking other clients.

_Compactness_: Any type can be wrapped with `data::Indirect`, which changes the encoding to a single offset. This is exactly the same encoding as `Option`, but can never be `None`.

_Deffered validation_: Any type can be wrapped with `data::Defer`, the defer wrapper postpones any validation, copying, and deserialization of that field. You must call `data::Defer::get(&self) -> data::Result`, which performs validation, copying, and deserialization. `data::Defer` has exactly the same layout as `T`, so it can be added and removed willy nilly, without breaking other clients.

_Repurposing fields_: This is tricky. If a field is marked `data::Ignore`, and old clients no longer exist which have the field as `T`, then new clients could repurpose it, but only if they do so with a new `T` which has the same layout as the old T, and accepts the default value that clients give it. This can be accomplished with the following sequence of field types: `Option` -> `data::Ignore>` -> `Option`. `Option` has the same layout as `Option`, and `None` is a valid value for both. Another possible solution is to add `data::Default`, and use the following sequence: `data::Default` -> `data::Ignore>` -> `data::Default`. `data::Default` would have the same encoding as `Option`, but must be given a default value with an annotation. So `0` would be a valid encoding that would provide the default value when decoded. This would give you the same ability to change types, but would avoid requiring client code to handle the options. The above two could be simplified by adding `data::Deprecated`. Deprecated is guaranteed to have the same layout as `Option` and `data::Default`, to always encode as `0`, and always be ignored when decoding. This would allow transitions like: `Option` -> `data::Deprecated` -> `data::Default` and avoid excessively messy types.