apache / apache/avro-rs

Feature request: `SchemaAwareResolvingDeserializer` — direct-to-type schema evolution without the intermediate `Value`

Open
#575 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
130
Forks
62
Avg merge
11h
Merged PRs (30d)
45

Description

# Summary

First of all, thank you very much for your work on `avro-rs`! The `SchemaAwareDeserializer` introduced in #512 is a great addition.

I would like to kindly ask about the status of the *resolving* counterpart to that deserializer. Today, `SchemaAwareDeserializer` decodes directly into `T` only when the reader schema matches the writer schema. As soon as a reader schema differs , the serde path deliberately bails out with a `todo!`/`panic!`, pointing at a not-yet-implemented `SchemaAwareResolvingDeserializer`.

I haven't found a tracking ticket for this.

# Where this shows up in the code

The gap is marked with three matching `TODO`s on `main`(pinned to `660c655cac0cdf3bc5c6818bf6b5e445eb09bdce`):

- `GenericDatumReader::read_deser`
[avro/src/reader/datum.rs#L149-L150](https://github.com/apache/avro-rs/blob/660c655cac0cdf3bc5c6818bf6b5e445eb09bdce/avro/src/reader/datum.rs#L149-L150)
```rust
// TODO: Implement SchemaAwareResolvingDeserializer
panic!("Schema aware deserialisation does not resolve schemas yet");
```
- `Reader` (object-container path)
[avro/src/reader/mod.rs#L134-L137](https://github.com/apache/avro-rs/blob/660c655cac0cdf3bc5c6818bf6b5e445eb09bdce/avro/src/reader/mod.rs#L134-L137)
```rust
// TODO: Implement SchemaAwareResolvingDeserializer
...
"Schema aware deserialisation does not resolve schemas yet"
```
- `BlockReader`
[avro/src/reader/block.rs#L233-L234](https://github.com/apache/avro-rs/blob/660c655cac0cdf3bc5c6818bf6b5e445eb09bdce/avro/src/reader/block.rs#L233-L234)
```rust
// TODO: Implement SchemaAwareResolvingDeserializer
panic!("Schema aware deserialisation does not resolve schemas yet");
```

The non-resolving deserializer itself lives in [avro/src/serde/deser_schema/mod.rs](https://github.com/apache/avro-rs/blob/660c655cac0cdf3bc5c6818bf6b5e445eb09bdce/avro/src/serde/deser_schema/mod.rs) and carries a single schema, doing no writer→reader resolution.

# Why it matters

Currently, schema evolution is only available on the `Value` route: decode to `Value` against the writer schema, then `Value::resolve*` against the reader schema, then optionally `from_value::()`. This allocates a full `Value` tree (and, without a cached `ResolvedSchema`, a fresh name map) per record — precisely the overhead the schema-aware deserializer was designed to avoid. A resolving deserializer would close that gap and, as noted in the #512 description, could eventually allow deprecating `from_value` / `to_value` entirely.

# Request

Would you be willing to share the implementation state of a `SchemaAwareResolvingDeserializer`?

I completely understand this is volunteer-maintained work and that priorities and timelines shift; please take this only as gentle interest, not pressure!

Thanks,

@flxo

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing implementation in avro/src/serde/deser_schema/mod.rs, then trace the TODO paths in avro/src/reader/datum.rs, avro/src/reader/mod.rs, and avro/src/reader/block.rs. Done means differing writer and reader schemas can deserialize directly into T through these paths without first constructing a Value, replacing the current unresolved-schema failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.