byte array has better performance than ByteBuffer
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
The BytePacker should add the following method
public abstract void unpack8Values(final byte[] input, final int inPos, final int[] output, final int outPos);
to replace method
@Deprecated
public void unpack8Values(final byte[] input, final int inPos, final int[] output, final int outPos) {
unpack8Values(ByteBuffer.wrap(input), inPos, output, outPos);
}
Tested by me byte array has better performance than ByteBuffer,
The test result is:
[Unpack8ValuesByteArray spent time] 80 ms
[Unpack8ValuesByteBuffer spent time] 133 ms
The test code is:
package org.apache.parquet.column.values.bitpacking;
import java.nio.ByteBuffer;
public class ByteBufferTest {
private static final BytePacker bytePacker = Packer.LITTLE_ENDIAN.newBytePacker(7);
private static final int COUNT = 100000;
public static void main(String[] args) {
byte [] in = new byte[1008];
int [] out = new int[1152];
int [] out1 = new int[1152];
int [] out2 = new int[1152];
int res = 0;
for(int i = 0; i < in.length; i++) {
in[i] = (byte) i;
}
for(int i = 0; i < COUNT; i++) {
res += unpack8ValuesBytes(in, out, i % out.length);
}
res = 0;
long t1 = System.currentTimeMillis();
for(int i = 0; i < COUNT; i++) {
res += unpack8ValuesBytes(in, out1, i % out.length);
}
long t2 = System.currentTimeMillis();
System.out.println("[Unpack8ValuesByteArray spent time] " + (t2-t1) + " ms");
ByteBuffer byteBuffer = ByteBuffer.wrap(in);
for(int i = 0; i < COUNT; i++) {
res += unpack8ValuesByteBuffer(byteBuffer, out, i % out.length);
}
res = 0;
long t3 = System.currentTimeMillis();
for(int i = 0; i < COUNT; i++) {
res += unpack8ValuesByteBuffer(byteBuffer, out2, i % out.length);
}
long t4 = System.currentTimeMillis();
System.out.println("[Unpack8ValuesByteBuffer spent time] " + (t4-t3) + " ms");
for (int i=0; i**Note**: *This issue was originally created as [PARQUET-2190](https://issues.apache.org/jira/browse/PARQUET-2190). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 BytePacker.unpack8Values 和入口点 Packer.LITTLE_ENDIAN.newBytePacker(7) 开始,然后检查提供当前 ByteBuffer 路径的具体 packer 实现。使用 ByteBufferTest 示例比较 byte-array 和 ByteBuffer 的等效结果与性能;当 byte-array API 能够在所有实现中工作且不改变解包后的值时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering, performance
- Issue 类型
- 功能
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 38/100