apache / apache/paimon

[Bug] parquet java version has a logging bug when debug logging is turned on

Open
#5,231 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

10.0

### Compute Engine

JavaAPI

### Minimal reproduce step

I'm just writing out data into s3 using a PostgreSQL catalog, and suddenly things start failing with a long stack trace which crashes out the thread:

```
2025-03-08T00:51:58.899543736Z ERROR Unable to format msg: 0, VisitedIndex{visitedIndexes={}}: [] r:0 java.lang.IllegalArgumentException: found 1 argument placeholders, but provided 0 for pattern `0, VisitedIndex{visitedIndexes={}}: [] r:0`
at org.apache.logging.log4j.message.ParameterFormatter.formatMessage(ParameterFormatter.java:248)
at org.apache.logging.log4j.message.ParameterizedMessage.formatTo(ParameterizedMessage.java:282)
at org.apache.logging.log4j.core.pattern.MessagePatternConverter$SimpleMessagePatternConverter.format(MessagePatternConverter.java:120)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:44)
at org.apache.logging.log4j.core.pattern.StyleConverter.format(StyleConverter.java:117)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:44)
at org.apache.logging.log4j.core.pattern.HighlightConverter.format(HighlightConverter.java:248)
at org.apache.logging.log4j.core.layout.PatternLayout$NoFormatPatternSerializer.toSerializable(PatternLayout.java:355)
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:252)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:238)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2906)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2859)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2841)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2631)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2394)
at org.apache.logging.slf4j.Log4jLogger.debug(Log4jLogger.java:128)
at org.apache.paimon.shade.org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.log(MessageColumnIO.java:283)
at org.apache.paimon.shade.org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.printState(MessageColumnIO.java:269)
at org.apache.paimon.shade.org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.startMessage(MessageColumnIO.java:294)
at org.apache.paimon.shade.org.apache.parquet.io.RecordConsumerLoggingWrapper.startMessage(RecordConsumerLoggingWrapper.java:175)
at org.apache.paimon.format.parquet.writer.ParquetRowDataWriter.write(ParquetRowDataWriter.java:71)
at org.apache.paimon.format.parquet.writer.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:75)
at org.apache.paimon.format.parquet.writer.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:56)
at org.apache.paimon.shade.org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:138)
at org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:103)
at org.apache.paimon.format.parquet.writer.ParquetBulkWriter.addElement(ParquetBulkWriter.java:47)
at org.apache.paimon.io.SingleFileWriter.writeImpl(SingleFileWriter.java:126)
at org.apache.paimon.io.StatsCollectingSingleFileWriter.write(StatsCollectingSingleFileWriter.java:84)
at org.apache.paimon.io.KeyValueDataFileWriter.write(KeyValueDataFileWriter.java:123)
at org.apache.paimon.io.KeyValueDataFileWriter.write(KeyValueDataFileWriter.java:58)
at org.apache.paimon.io.RollingFileWriter.write(RollingFileWriter.java:80)
at org.apache.paimon.mergetree.SortBufferWriteBuffer.forEach(SortBufferWriteBuffer.java:162)
at org.apache.paimon.mergetree.MergeTreeWriter.flushWriteBuffer(MergeTreeWriter.java:225)
at org.apache.paimon.mergetree.MergeTreeWriter.prepareCommit(MergeTreeWriter.java:264)
at org.apache.paimon.operation.AbstractFileStoreWrite.prepareCommit(AbstractFileStoreWrite.java:218)
at org.apache.paimon.operation.MemoryFileStoreWrite.prepareCommit(MemoryFileStoreWrite.java:155)
at org.apache.paimon.table.sink.TableWriteImpl.prepareCommit(TableWriteImpl.java:253)
at com.toro.trc.services.confluence.utils.TAvroToPaimon.writeToTable(TAvroToPaimon.java:137)
at com.toro.trc.services.confluence.sinks.TPaimonTableSink.publish(TPaimonTableSink.java:70)
at com.toro.trc.services.confluence.sinks.TRobotPulsarPaimonSink.publish(TRobotPulsarPaimonSink.java:51)
at com.toro.trc.services.confluence.TConflux.writeToSink(TConflux.java:146)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)
```

### What doesn't meet your expectations?

The thread crashes because of a log4j issue inside of the parquet library. Obviously not ideal.

### Anything else?

Took a little bit, but I found that this is a known issue with parquet-java that has since been fixed in this PR: https://github.com/apache/parquet-java/pull/1294

The first release to include this fix was 1.14.0. So I assume just upgrading to at least that version should address the issue. Latest version is 1.15.0. Would it be reasonable to just move to the latest version?

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the Parquet dependency declaration and the entry points named in ParquetRowDataWriter.java and ParquetRowDataBuilder.java, then compare the bundled parquet-java version with the fix in PR 1294. Verify the change with a debug-logging write to the PostgreSQL-catalog/S3 path and confirm the thread no longer crashes from the Log4j formatting error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.