Producer encoding errors: silent data loss and type mismatch
- Dominant language
- Rust
- Stars
- 46
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Two related issues with how producer encoding errors are handled:
### 1. Silent data loss in fanout
In `src/streams/producer/worker.rs` (~line 224), when `Datum::encode()` fails during fanout, the error is logged and the datum is dropped. The caller of `producer.send()` has no way to know this happened.
Additionally, the dropped datum may be routed to the `undelivered` sink (~line 248), which is designed for "no consumers matched" — not "serialization failed." These two failure modes are conflated.
### 2. Error type hardcodes postcard
In `src/streams/producer/error.rs`, the `Encoding` variant uses the concrete `EncodeError` (postcard):
```rust
use crate::primitives::EncodeError;
// ...
Encoding(D, EncodeError),
```
But the `Datum` trait allows custom `EncodeError` types via associated types. The `Clone` impl reconstructs the error via `EncodeError::custom(e.to_string())`, which is lossy for non-postcard error types.
This breaks the abstraction that the customizable encoding feature was meant to provide.
## Suggestion
- Propagate encoding errors back to the `send()` caller, or provide an explicit error callback
- Keep the `undelivered` sink for "no match" only; handle encoding failures separately
- Make `Error` generic over `D::EncodeError` (or use a boxed error) instead of hardcoding postcard
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.