apache / apache/parquet-java

BytesWritable is not in parquet 's writePrimitive method in DataWritableWriter class

Open
#1,715 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

I have 2 int fields and 1 string field . To insert it into parquet File I use 2 IntWritable and 1 BinaryWritable object .

For next record i can reuse my 2 IntWritable objects but i have to create a new BinaryWritable object because BinaryWritable do not have set method . This is a bottleneck for me in performance as i am doing new again for every nextrecord .

How do you guys recommend i solve this since BytesWritable is not in parquet 's writePrimitive in DataWritableWriter class . it has ByteWritable and we need BytesWritable

**Reporter**: [Manish Agarwal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=manish.agarwal)

**Note**: *This issue was originally created as [PARQUET-163](https://issues.apache.org/jira/browse/PARQUET-163). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading DataWritableWriter.writePrimitive and its handling of ByteWritable, then compare it with the BytesWritable and BinaryWritable usage described in the issue. Check whether existing writer tests cover writable binary values; done should be defined as supporting reusable BytesWritable input without creating a new binary object for every record.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.