linkedin / linkedin/avro-util

Allow user-specified functional transforms to be applied during deserialization.

Open
#398 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
87
Forks
73
Avg merge
2h 50m
Merged PRs (30d)
1

Description

As an extension of [issue 397](https://github.com/linkedin/avro-util/issues/397), allow users to specify functional transforms that, ideally, will be applied during deserialization. This allows users to specify what is most-optimal for their use case. Why is this valuable?

1. Records may be retrieved from data storage full of low cardinality duplicate strings. Interning these straight into the record can save memory usage.
2. Records that end up cached should ideally use immutable collections. Allowing user to do this on deserialization can save memory allocation overhead.
3. As specified in issue 397, utilization of specific libraries like fastutil could be a simple deserialization transform that the user configures.
...

I'm sure there are other reasons users would have. Currently my project takes Avro records and re-processes them after deserialization leading to wasted memory allocation overhead.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing issue 397 and the deserialization flow in avro-util, since no specific files or tests are named here. Define how users would configure functional transforms and where they should run during deserialization. Done means user-specified transforms can support cases such as string interning, immutable collections, or fastutil-backed values without a separate reprocessing pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.