Provide a ByteBuffer based write path
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Similar to the effort to use a ByteBuffer in the read path, we should provide a path to use a ByteBuffer in the write path.
The proposed idea is to provide an allocator in the writer that is passed down to CapacityByteArrayOutputStream and is used to allocate a ByteBuffer instead of the byte array that it currently uses.
I've attached a patch based on the 1.5 release that does that. There are two implementations of the allocator - one for Heap memory and one for Direct memory. In addition, the allocator interface allows for a 'release' method so that reference counted memory allocators like Netty's ByteBuf can be used to provide an allocator.
**Reporter**: [Parth Chandra](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=parthc) / @parthchandra
#### Related issues:
- [Move off of Parquet fork](https://issues.apache.org/jira/browse/DRILL-1410) (is depended upon by)
**Note**: *This issue was originally created as [PARQUET-78](https://issues.apache.org/jira/browse/PARQUET-78). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the existing ByteBuffer-based read path with the writer and CapacityByteArrayOutputStream named in the issue. Trace how the proposed allocator would supply heap or direct buffers and how release would be handled, then verify that the write path supports both allocator implementations and reference-counted memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100