apache / apache/hudi

[BUG] PostgresDebeziumAvroPayload throws ClassCastException on BYTES columns after lazy Avro deserialization

Open
#19,563 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:cdc priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem you faced

Merging a Debezium Postgres change event that carries a non-null `BYTES` column (or a union containing `BYTES`) fails with a `ClassCastException`:

```
java.lang.ClassCastException: org.apache.hudi.common.model.SerializableIndexedRecord
cannot be cast to org.apache.avro.generic.GenericData$Record
at org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload.containsBytesToastedValues(PostgresDebeziumAvroPayload.java:143)
```

`PostgresDebeziumAvroPayload` casts records to the concrete class `GenericData.Record` in `containsBytesToastedValues` and in `mergeToastedValuesIfPresent`. Since #13987 ("perf: Lazy deserialization of Avro indexed record"), `BaseAvroPayload.getRecord` returns a `SerializableIndexedRecord` whenever the payload's record does not already carry the exact `Schema` instance being merged with, which is the ordinary case in production and after any Kryo round trip. `SerializableIndexedRecord` implements `GenericRecord` but does not extend `GenericData.Record`, so the cast fails.

That PR converted the casts in `containsStringToastedValues` to the `GenericRecord` interface but left the two in `containsBytesToastedValues` and one in `mergeToastedValuesIfPresent`, which is why only tables with a `BYTES` column are affected.

The existing `TestPostgresDebeziumAvroPayload#testMergeWithToastedValues` misses this because it constructs the record and merges with the very same `Schema` instance, so the payload hands back the original `GenericData.Record` and never takes the lazy path.

### To Reproduce

1. Ingest a Postgres table through the Debezium source with `hoodie.datasource.write.payload.class=org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload`.
2. Give the table a `BYTES` (or nullable `BYTES`) column with a non-null value.
3. Apply an update whose LSN is ahead of the record already in storage, so the incoming event wins the LSN comparison and the TOAST check runs.

### Expected behavior

The TOAST check works against any `GenericRecord` implementation, and the merge preserves the previous value for toasted columns.

### Environment Description

* Hudi version: master (`c63c9bfa79f` and later)

### Additional context

Affects any `IndexedRecord` implementation that is not a `GenericData.Record`, so the same latent problem exists on `branch-0.x` even though the lazy-deserialization path that triggers it is only on master.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.