linkedin / linkedin/avro-util

Consider adopting fastutil for deserialization of primitive collections

Open
#397 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
87
Forks
73
Avg merge
2h 50m
Merged PRs (30d)
1

Description

Array and Maps of primitive types are common in Avro schema. Java collections do an acceptable generic job, but type-specific collections such as [fastutil](https://fastutil.di.unimi.it/) can save overhead in multiple ways:

1. No boxing/unboxing of numbers.
2. Primitive type-specific collections lower memory overhead of underlying representations.
3. Which is more pronounced in JVM heaps over 32gb due to 16bit object reference overhead.

fastutil is well respected and widely adopted for performance, so using it for Java representation of Avro data could provide a nice benefit.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the Java deserialization code for Avro arrays and maps, then determine the scope and compatibility requirements for adopting fastutil; done would require an agreed design and implementation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.