apache / apache/arrow-java

[Java] Slow LZ4 compression using java Arrow 12.0.0

未关闭
#192 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: bug
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

500 MB of arrow data took a few hours to compress. A 5GB selection ran for a couple of days and did not complete. The one that did worked fine once it was done and I was able to read it back in and use it. Without compression, data writes out fine in a much shorter time frame for both. Using the following code to set up my writer:
```
ArrowFileWriter writer = new ArrowFileWriter(root, null, Channels.newChannel(fos), new HashMap<>(), IpcOption.DEFAULT, CommonsCompressionFactory.INSTANCE, CompressionUtil.CodecType.LZ4_FRAME));
```
Running code with:
```
java -version
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment GraalVM CE 22.3.2 (build 17.0.7+7-jvmci-22.3-b18)
OpenJDK 64-Bit Server VM GraalVM CE 22.3.2 (build 17.0.7+7-jvmci-22.3-b18, mixed mode, sharing)
```

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

首先,使用提供的 Java 和 OpenJDK 版本,通过 ArrowFileWriter、CommonsCompressionFactory 和 CompressionUtil.CodecType.LZ4_FRAME 重现报告的压缩行为。比较报告数据大小下的压缩和未压缩写入时间;当确定 LZ4 运行时间过长的原因并验证问题已解决后,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
performance
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。