apache / apache/paimon

[Bug] CDC MySQL data to Paimon using PaimonAction. Exception: AvroRuntimeException: Unrecognized codec: snappy

Open
#5,395 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

Paimon(on oss): 1.0.1

### Compute Engine

Flink: 1.20.1

### Minimal reproduce step

The cdc data binlog was interrupted abnormally at night. After restarting the task the next morning (new task, not restarting from checkpoints/savepoints, because the binlog file was lost), the data could not be written. The exception was always thrown: org.apache.paimon.shade.org.apache.avro.AvroRuntimeException: Unrecognized codec: snappy

### What doesn't meet your expectations?

2025-04-03 11:31:23
java.io.IOException: org.apache.paimon.shade.org.apache.avro.AvroRuntimeException: Unrecognized codec: snappy
at org.apache.paimon.flink.sink.cdc.CdcRecordStoreMultiWriteOperator.processElement(CdcRecordStoreMultiWriteOperator.java:171)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.paimon.shade.org.apache.avro.AvroRuntimeException: Unrecognized codec: snappy
at org.apache.paimon.shade.org.apache.avro.file.CodecFactory.fromString(CodecFactory.java:159)
at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.resolveCodec(DataFileStream.java:158)
at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:144)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.(DataFileReader.java:143)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.(DataFileReader.java:134)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:74)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.createReaderFromPath(AvroBulkFormat.java:82)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.(AvroBulkFormat.java:68)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.(AvroBulkFormat.java:55)
at org.apache.paimon.format.avro.AvroBulkFormat.createReader(AvroBulkFormat.java:52)
at org.apache.paimon.utils.FileUtils.createFormatReader(FileUtils.java:130)
at org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:167)
at org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:140)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:123)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:97)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:93)
at org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:86)
at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:403)
at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:394)
at org.apache.paimon.operation.AbstractFileStoreScan.doPlan(AbstractFileStoreScan.java:276)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:225)
at org.apache.paimon.operation.AbstractFileStoreWrite.scanExistingFileMetas(AbstractFileStoreWrite.java:441)
at org.apache.paimon.operation.AbstractFileStoreWrite.createWriterContainer(AbstractFileStoreWrite.java:398)
at org.apache.paimon.operation.AbstractFileStoreWrite.lambda$getWriterWrapper$2(AbstractFileStoreWrite.java:373)
at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
at org.apache.paimon.operation.AbstractFileStoreWrite.getWriterWrapper(AbstractFileStoreWrite.java:372)
at org.apache.paimon.operation.AbstractFileStoreWrite.write(AbstractFileStoreWrite.java:139)
at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:175)
at org.apache.paimon.flink.sink.StoreSinkWriteImpl.write(StoreSinkWriteImpl.java:180)
at org.apache.paimon.flink.sink.cdc.CdcRecordStoreMultiWriteOperator.processElement(CdcRecordStoreMultiWriteOperator.java:169)
... 13 more

### Anything else?

When CDC synchronizes data, other tasks also write data. This may be the reason for the exception.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.paimon.format.avro.AvroBulkFormat and the shaded Avro CodecFactory path shown in the stack trace, then inspect how the CDC write reaches CdcRecordStoreMultiWriteOperator. Reproduce the restart scenario with Flink 1.20.1 and Paimon 1.0.1; done means the CDC task writes data without the Unrecognized codec: snappy exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.