[Bug] CDC MySQL data to Paimon using PaimonAction. Exception: AvroRuntimeException: Unrecognized codec: snappy
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
Paimon(on oss): 1.0.1
### Compute Engine
Flink: 1.20.1
### Minimal reproduce step
The cdc data binlog was interrupted abnormally at night. After restarting the task the next morning (new task, not restarting from checkpoints/savepoints, because the binlog file was lost), the data could not be written. The exception was always thrown: org.apache.paimon.shade.org.apache.avro.AvroRuntimeException: Unrecognized codec: snappy
### What doesn't meet your expectations?
2025-04-03 11:31:23
java.io.IOException: org.apache.paimon.shade.org.apache.avro.AvroRuntimeException: Unrecognized codec: snappy
at org.apache.paimon.flink.sink.cdc.CdcRecordStoreMultiWriteOperator.processElement(CdcRecordStoreMultiWriteOperator.java:171)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.paimon.shade.org.apache.avro.AvroRuntimeException: Unrecognized codec: snappy
at org.apache.paimon.shade.org.apache.avro.file.CodecFactory.fromString(CodecFactory.java:159)
at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.resolveCodec(DataFileStream.java:158)
at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:144)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.(DataFileReader.java:143)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.(DataFileReader.java:134)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:74)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.createReaderFromPath(AvroBulkFormat.java:82)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.(AvroBulkFormat.java:68)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.(AvroBulkFormat.java:55)
at org.apache.paimon.format.avro.AvroBulkFormat.createReader(AvroBulkFormat.java:52)
at org.apache.paimon.utils.FileUtils.createFormatReader(FileUtils.java:130)
at org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:167)
at org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:140)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:123)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:97)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:93)
at org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:86)
at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:403)
at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:394)
at org.apache.paimon.operation.AbstractFileStoreScan.doPlan(AbstractFileStoreScan.java:276)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:225)
at org.apache.paimon.operation.AbstractFileStoreWrite.scanExistingFileMetas(AbstractFileStoreWrite.java:441)
at org.apache.paimon.operation.AbstractFileStoreWrite.createWriterContainer(AbstractFileStoreWrite.java:398)
at org.apache.paimon.operation.AbstractFileStoreWrite.lambda$getWriterWrapper$2(AbstractFileStoreWrite.java:373)
at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
at org.apache.paimon.operation.AbstractFileStoreWrite.getWriterWrapper(AbstractFileStoreWrite.java:372)
at org.apache.paimon.operation.AbstractFileStoreWrite.write(AbstractFileStoreWrite.java:139)
at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:175)
at org.apache.paimon.flink.sink.StoreSinkWriteImpl.write(StoreSinkWriteImpl.java:180)
at org.apache.paimon.flink.sink.cdc.CdcRecordStoreMultiWriteOperator.processElement(CdcRecordStoreMultiWriteOperator.java:169)
... 13 more
### Anything else?
When CDC synchronizes data, other tasks also write data. This may be the reason for the exception.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.paimon.format.avro.AvroBulkFormat and the shaded Avro CodecFactory path shown in the stack trace, then inspect how the CDC write reaches CdcRecordStoreMultiWriteOperator. Reproduce the restart scenario with Flink 1.20.1 and Paimon 1.0.1; done means the CDC task writes data without the Unrecognized codec: snappy exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100