Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Error when using zstd compression with parquet output format

未關閉
#257 1 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
GCS
主要語言
Java
星號
58
分支
39
平均合併
2 天 10 小時
30 天內合併 PR
5

描述

Could this be because of this: https://issues.apache.org/jira/browse/PARQUET-1866 maybe?

```log
2022-12-01 00:56:14,469 ERROR [kafka-sink-connector-gcs|task-0] WorkerSinkTask{id=kafka-sink-connector-gcs-0} Commit of offsets threw an unexpected exception for sequence number 6: null (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-kafka-sink-connector-gcs-0]
org.apache.kafka.connect.errors.ConnectException: java.lang.RuntimeException: native zStandard library not available: this version of libhadoop was built without zstd support.
at io.aiven.kafka.connect.gcs.GcsSinkTask.flushFile(GcsSinkTask.java:127)
at java.base/java.util.HashMap.forEach(HashMap.java:1337)
at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1505)
at io.aiven.kafka.connect.gcs.GcsSinkTask.flush(GcsSinkTask.java:110)
at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:125)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:404)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:374)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:218)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:203)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:244)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: native zStandard library not available: this version of libhadoop was built without zstd support.
at org.apache.hadoop.io.compress.ZStandardCodec.checkNativeCodeLoaded(ZStandardCodec.java:65)
at org.apache.hadoop.io.compress.ZStandardCodec.getCompressorType(ZStandardCodec.java:153)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.(CodecFactory.java:144)
at org.apache.parquet.hadoop.CodecFactory.createCompressor(CodecFactory.java:206)
at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:189)
at org.apache.parquet.hadoop.ParquetWriter.(ParquetWriter.java:287)
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:564)
at io.aiven.kafka.connect.common.output.parquet.ParquetOutputWriter.writeRecords(ParquetOutputWriter.java:70)
at io.aiven.kafka.connect.gcs.GcsSinkTask.flushFile(GcsSinkTask.java:125)
... 15 more
```

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。