apache / apache/hudi

test automatic compression failed, no parquet file generated

Open
#8,964 7 comments 0 reactions 1 assignee Claimed by @danny0405 View on GitHub
issue:class-conflict priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**To Reproduce**

Steps to reproduce the behavior:

1.**Expected behavior**
1)behavior
```2023-06-14 13:45:30,484 INFO org.apache.hadoop.io.compress.CodecPool [] - Got brand-new compressor [.gz]
2023-06-14 13:45:30,922 INFO org.apache.hudi.io.HoodieCreateHandle [] - New CreateHandle for partition : with fileId 1530c350-1cc7-474b-b36a-2089799a2ea5
2023-06-14 13:45:30,928 INFO org.apache.hudi.io.HoodieCreateHandle [] - Closing the file 1530c350-1cc7-474b-b36a-2089799a2ea5 as we are done with all the records 3
2023-06-14 13:45:30,941 ERROR org.apache.hudi.sink.compact.CompactFunction [] - Executor executes action [Execute compaction for instant 20230606150932822 from task 0] error
java.lang.NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B
at org.apache.parquet.format.PageHeader.setUncompressed_page_sizeIsSet(PageHeader.java:340) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.format.PageHeader.(PageHeader.java:230) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.format.converter.ParquetMetadataConverter.newDataPageHeader(ParquetMetadataConverter.java:1746) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.format.converter.ParquetMetadataConverter.writeDataPageV1Header(ParquetMetadataConverter.java:1828) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:186) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.io.storage.HoodieAvroParquetWriter.close(HoodieAvroParquetWriter.java:84) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.io.HoodieCreateHandle.close(HoodieCreateHandle.java:211) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleInsert(HoodieFlinkCopyOnWriteTable.java:419) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:231) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.sink.compact.CompactFunction.doCompaction(CompactFunction.java:110) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.sink.compact.CompactFunction.lambda$processElement$0(CompactFunction.java:95) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_301]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
2023-06-14 13:45:30,946 INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading HoodieTableMetaClient from hdfs:///flink_hudi/hudi_shipments
```

2) use example
```create table hudi_shipments(
shipment_id INT,
order_id INT,
origin STRING,
destination STRING,
is_arrived BOOLEAN,
PRIMARY KEY (shipment_id) NOT ENFORCED
)
with(
'connector'='hudi',
'path'='hdfs:///flink_hudi/hudi_shipments',
'hoodie.datasource.write.recordkey.field'='id',
'hoodie.parquet.max.file.size'='268435456',
'write.tasks'='4',
'write.bucket_assign.tasks'='1',
'write.task.max.size'='1024',
'write.rate.limit'='30000',
'table.type'='MERGE_ON_READ',
'compaction.tasks'='1',
'compaction.async.enabled'='true',
'compaction.delta_commits'='5',
'compaction.max_memory'='1024',
'compaction.schedule.enabled' = 'true',
'compaction.trigger.strategy' = 'num_commits',
'changelog.enabled' = 'true',
'read.streaming.enabled' = 'true',
'read.streaming.skip_compaction' = 'true'
);
insert into hudi_shipments select* from shipments;
```

2.**Environment Description**
* Hudi version :
0.12.0
* Hadoop version :
CDH5.13.3
* Storage (HDFS/S3/GCS..) :
HDFS
* Running on Docker? (yes/no) :
no
* flink
1.14

3.**Additional context**
Manual compression successful, automatic compression failed

4.**Stacktrace**

Heap and stack have no memory overflow

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.