apache / apache/avro-rs

Serialization performance tuning how-to?

Open
#322 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
130
Forks
62
Avg merge
11h
Merged PRs (30d)
45

Description

I'm working through writing a service that downloads `n` files concurrently - then writes them to a single avro file.
- Is there a recommended way to parallel serialize data? I see `Writer` calls `maybe_write_header` in all public append APIs, along with `into_inner`, which makes it hard to just get the raw bytes of serialized rows without the header attached. It doesn't look like the raw `Serializer` impl in `ser.rs` is public either. What is the recommended way to split serialization work across cores?
- Schema validation per-value appended is expensive - it would be really nice to have compile flags around it so it can be stripped out for production, or have a sampling rate attached to it to retain some runtime safety?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Writer's maybe_write_header and into_inner calls, then read the raw Serializer implementation in ser.rs. Determine whether the requested header-free parallel serialization and configurable schema validation need an API or design change. Done means an agreed, scoped approach with tests covering the selected serialization and validation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.