Serialization performance tuning how-to?
- Dominant language
- Rust
- Stars
- 130
- Forks
- 62
- Avg merge
- 11h
- Merged PRs (30d)
- 45
Description
I'm working through writing a service that downloads `n` files concurrently - then writes them to a single avro file.
- Is there a recommended way to parallel serialize data? I see `Writer` calls `maybe_write_header` in all public append APIs, along with `into_inner`, which makes it hard to just get the raw bytes of serialized rows without the header attached. It doesn't look like the raw `Serializer` impl in `ser.rs` is public either. What is the recommended way to split serialization work across cores?
- Schema validation per-value appended is expensive - it would be really nice to have compile flags around it so it can be stripped out for production, or have a sampling rate attached to it to retain some runtime safety?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Writer's maybe_write_header and into_inner calls, then read the raw Serializer implementation in ser.rs. Determine whether the requested header-free parallel serialization and configurable schema validation need an API or design change. Done means an agreed, scoped approach with tests covering the selected serialization and validation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100