apache / apache/parquet-java

Benchmark the assembly of thrift objects, and possibly create a more efficient ReplayingTProtocol

Open
#1,391 2 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Minor Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The current implementation of parquet thrift creates an instance of TProtocol for each value of each record and builds a stack of these events, which are then replayed back to the TBase.

I'd be curious to benchmark this, and if it's slow, try building a "ReplayingTProtocol" that instead of having a stack of TProtocol instances, contains a primitive array of each type. As events are fed into this replaying TProtocol, it would just add these primitives to its buffers, and then the TBase would drain them. This would effectively let us stream the values into the TBase without making an object allocation for each value.

The buffers could be set to a certain size, and if they fill up (which they sholdn't in most cases), the TBase could begin draining the protocol until it is empty again, at which point the TProtocol can block the TBase from draining further while the parque record assembly feeds it more events.

This is all moot if it turns out not to be bottleneck though :)

**Reporter**: [Alex Levenson](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=alexlevenson) / @isnotinvain

**Note**: *This issue was originally created as [PARQUET-33](https://issues.apache.org/jira/browse/PARQUET-33). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the current parquet thrift record-assembly path that creates a TProtocol per value and benchmark it before designing changes. Compare allocation and throughput with the existing event-stack replay; done means the benchmark supports a clear decision about whether a ReplayingTProtocol is warranted, with scope and tests documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.