BytesWritable is not in parquet 's writePrimitive method in DataWritableWriter class
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
I have 2 int fields and 1 string field . To insert it into parquet File I use 2 IntWritable and 1 BinaryWritable object .
For next record i can reuse my 2 IntWritable objects but i have to create a new BinaryWritable object because BinaryWritable do not have set method . This is a bottleneck for me in performance as i am doing new again for every nextrecord .
How do you guys recommend i solve this since BytesWritable is not in parquet 's writePrimitive in DataWritableWriter class . it has ByteWritable and we need BytesWritable
**Reporter**: [Manish Agarwal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=manish.agarwal)
**Note**: *This issue was originally created as [PARQUET-163](https://issues.apache.org/jira/browse/PARQUET-163). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading DataWritableWriter.writePrimitive and its handling of ByteWritable, then compare it with the BytesWritable and BinaryWritable usage described in the issue. Check whether existing writer tests cover writable binary values; done should be defined as supporting reusable BytesWritable input without creating a new binary object for every record.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100