apache / apache/avro-rs

Tracking issue: Serde and `#[derive(AvroSchema)]` improvements

Open
#361 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
130
Forks
62
Avg merge
11h
Merged PRs (30d)
45

Description

Our current Serde and derive implementation works, but has room for improvement. This is a unordered list of improvements I'm planning to work on.

- [x] Add documentation for the derive macro (https://github.com/apache/avro-rs/tree/feat/improve_documentation)
- [x] All supported attributes
- [x] Compatibility and interaction with Serde
- [ ] Make `avro(default)` (no specific value) use `Default::::default()` instead of always using `None`
- [ ] Add `append_ser` support to `GenericSingleObjectWriter`/`to_avro_datum`
- [x] Move all Serde related documentation and functions to the `serde` module (and make that public), so the main documentation is less cluttered (https://github.com/apache/avro-rs/tree/feat/improve_documentation)
- [x] Also add a short explainer of how Avro works to the Serde part
- [ ] Schema aware deserialisation (see #237)
- [ ] Implement `SerializeMap::serialize_entry`
- [ ] Allow disabling getting the rustdoc via `doc = false`
- [ ] In the derived `get_schema_with_ctxt` validate that the schema is valid (https://github.com/apache/avro-rs/issues/365#issuecomment-3730627801)
- [x] Support newtype structs
- [x] Support tuples
- [x] Make clear that errors come from AvroSchema, as that is not visible to the user
- [x] Support `char` (as a `Schema::String` or a `Schema::Fixed(4)`)
- [ ] Cleanup derive tests to be more like `avro_3687_basic_enum_with_default` instead of `test_avro_3709_record_field_attributes`
- [ ] Implement `AvroSchemaComponent` for all types that `Serde` implements [`Serialize`](https://docs.rs/serde/latest/serde/trait.Serialize.html) for
- [x] Allow overwriting the schema type for a field (i.e. `avro(type = "bytes")`, see #350) (#397)
- [x] Support `#[serde(transparent)]` (#398)
- [x] Make `AvroSchemaComponent` always available (#394)
- [x] Try to use as many Serde attributes as possible so that user doesn't have duplicate attributes (`avro` + `serde`) (#377)
- [x] Check Serde attributes so we can error on incompatible attributes (i.e. `skip_serializing_if` without a `avro(default)` (#377)

Future work:
- [ ] Support simple closures for the `with` attribute (i.e. `#[avro(with = || Schema::Bytes)]`)

Not possible:
- [ ] Use LazyLock to only create the schema once (inspired by https://github.com/lerouxrgd/rsgen-avro/pull/90)
- This does not work for types with generics or lifetimes, as the `const`/`static` would be shared between all implementations
- It also doesn't work for types without generics or lifetimes as the output can change based on the `names` and `enclosing_namespace` arguments
- [ ] Add a test for every derive that runs `T::get_schema()`
- This does not work for types with generics or lifetimes, we don't know what bounds are required on the types so we can't name the type.
- This also doesn't work for types without generics of lifetimes, because derives are only allowed to emit `impl` blocks, so we can't emit a test function

Contributor guide

No contributing guide indexed for this repository

Research direction

This is a broad tracking issue with several unchecked Serde and derive improvements rather than one scoped task. Start by choosing a specific item, then read the public serde module, get_schema_with_ctxt, and the mentioned derive tests avro_3687_basic_enum_with_default and test_avro_3709_record_field_attributes. Done depends on the selected checklist item and its corresponding tests passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.