apache / apache/hudi

maxCapacityHint can't be less than initialSlabSize 64 1

Open
#14,919 1 comment 0 reactions 1 assignee Claimed by @cshuo View on GitHub
area:table-service component:compaction engine:flink priority:medium type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

flink: 1.13.3

hudi: 0.10.0-SNAPSHOT

hive: 2.1.1

I found the following exception when testing HoodieFlinkStreamer。

I tracked the code and found that the reason is that the default value given by the Flink WRITE_PARQUET_PAGE_SIZE parameter is 1, but the CapacityByteArrayOutputStream constructor requires no less than the default value of 64。

Why the default value of WRITE_PARQUET_PAGE_SIZE is 1, can I fix it?
{code:java}
java.lang.IllegalArgumentException: maxCapacityHint can't be less than initialSlabSize 64 1
    at org.apache.parquet.Preconditions.checkArgument(Preconditions.java:53)
    at org.apache.parquet.bytes.CapacityByteArrayOutputStream.(CapacityByteArrayOutputStream.java:153)
    at org.apache.parquet.column.values.rle.RunLengthBitPackingHybridEncoder.(RunLengthBitPackingHybridEncoder.java:125)
    at org.apache.parquet.column.values.rle.RunLengthBitPackingHybridValuesWriter.(RunLengthBitPackingHybridValuesWriter.java:34)
    at org.apache.parquet.column.ParquetProperties.newColumnDescriptorValuesWriter(ParquetProperties.java:131)
    at org.apache.parquet.column.ParquetProperties.newDefinitionLevelWriter(ParquetProperties.java:123)
    at org.apache.parquet.column.impl.ColumnWriterV1.createDLWriter(ColumnWriterV1.java:47)
    at org.apache.parquet.column.impl.ColumnWriterBase.(ColumnWriterBase.java:65)
    at org.apache.parquet.column.impl.ColumnWriterV1.(ColumnWriterV1.java:37)
    at org.apache.parquet.column.impl.ColumnWriteStoreV1.createColumnWriter(ColumnWriteStoreV1.java:41)
    at org.apache.parquet.column.impl.ColumnWriteStoreBase.(ColumnWriteStoreBase.java:94)
    at org.apache.parquet.column.impl.ColumnWriteStoreV1.(ColumnWriteStoreV1.java:30)
    at org.apache.parquet.column.ParquetProperties.newColumnWriteStore(ParquetProperties.java:180)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.initStore(InternalParquetRecordWriter.java:106)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.(InternalParquetRecordWriter.java:96)
    at org.apache.parquet.hadoop.ParquetWriter.(ParquetWriter.java:292)
    at org.apache.parquet.hadoop.ParquetWriter.(ParquetWriter.java:227)
    at org.apache.hudi.io.storage.HoodieParquetWriter.(HoodieParquetWriter.java:56)
    at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:76)
    at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:63)
    at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:49)
    at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:101)
    at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:67)
    at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:115)
    at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleInsert(HoodieFlinkCopyOnWriteTable.java:390)
    at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:201)
    at org.apache.hudi.sink.compact.CompactFunction.doCompaction(CompactFunction.java:106)
    at org.apache.hudi.sink.compact.CompactFunction.lambda$processElement$0(CompactFunction.java:94)
    at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2835
- Type: Bug
- Affects version(s):
- 0.10.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.