Parquet Java Serialization is very slow
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
Hi,
I am doing a POC to compare different data formats and its performance in terms of serialization/deserialization speed, storage size, compatibility between different language etc.
When I try to serialize a simple java object to parquet file, it takes _\*6-7 seconds\*_ vs same object's serialization to JSON is **_100 milliseconds._**
Could you help me to resolve this issue?
+**My Configuration and code snippet:**
**Gradle dependencies**
dependencies
{ compile group: 'org.springframework.boot', name: 'spring-boot-starter' compile group: 'org.projectlombok', name: 'lombok', version: '1.16.6' compile group: 'com.amazonaws', name: 'aws-java-sdk-bundle', version: '1.11.271' compile group: 'org.apache.parquet', name: 'parquet-avro', version: '1.10.0' compile group: 'org.apache.hadoop', name: 'hadoop-common', version: '3.1.1' compile group: 'org.apache.hadoop', name: 'hadoop-aws', version: '3.1.1' compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.1.1' compile group: 'joda-time', name: 'joda-time' compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.6.5' compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-joda', version: '2.6.5' }
**Code snippet:**+
public void serialize(List inputDataToSerialize, CompressionCodecName compressionCodecName) throws IOException {
Path path = new Path("s3a://parquetpoc/data_"+compressionCodecName+".parquet");
Path path1 = new Path("/Downloads/data_"+compressionCodecName+".parquet");
Class clazz = inputDataToSerialize.get(0).getClass();
try (ParquetWriter writer = **AvroParquetWriter.**builder(path1)
.withSchema(ReflectData.AllowNull.get().getSchema(clazz)) // generate nullable fields
.withDataModel(ReflectData.get())
.withConf(parquetConfiguration)
.withCompressionCodec(compressionCodecName)
.withWriteMode(OVERWRITE)
.withWriterVersion(ParquetProperties.WriterVersion.PARQUET_2_0)
.build()) {
for (D input : inputDataToSerialize)
{ writer.write(input); }
}
}
+**Model Used:**
@Data
public class Employee
{ //private UUID id; private String name; private int age; private Address address; }
@Data
public class Address
{ private String streetName; private String city; private Zip zip; }
@Data
public class Zip
{ private int zip; private int ext; }
private List **getInputDataToSerialize**(){
Address address = new Address();
address.setStreetName("Murry Ridge Dr");
address.setCity("Murrysville");
Zip zip = new Zip();
zip.setZip(15668);
zip.setExt(1234);
address.setZip(zip);
List employees = new ArrayList<>();
IntStream.range(0, 100000).forEach(i->{
Employee employee = new Employee();
// employee.setId(UUID.randomUUID());
employee.setAge(20);
employee.setName("Test"+i);
employee.setAddress(address);
employees.add(employee);
});
return employees;
}
**Note:**
**I have tried to save the data into local file system as well as AWS S3, but both are having same result - very slow.**
**Reporter**: [Felix Kizhakkel Jose](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=FelixKJose) / @FelixKJose
**Note**: *This issue was originally created as [PARQUET-1680](https://issues.apache.org/jira/browse/PARQUET-1680). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
まず、提供された Employee モデルと 100,000 件のレコード入力を使用して、AvroParquetWriter.builder、ReflectData.AllowNull.get().getSchema(clazz)、および writer.write 周辺のベンチマークを再現します。ローカルパスと S3 パスで、スキーマ生成、writer の構築、書き込みを比較します。支配的なレイテンシの原因を特定し、測定による改善を示せれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- aws, hadoop, java
- 領域
- data-engineering, performance
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100