apache / apache/doris

QuantileState protobuf serialization corrupts data in write/read_column_to_pb

Open
#64,149 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Version

trunk (4b7a09ff6fd)

### What's Wrong?

`DataTypeQuantileStateSerDe::write_column_to_pb` and `read_column_from_pb` use raw memory copy (`get_data_at` / `insert_data`) to serialize/deserialize `QuantileState` values. However, `QuantileState` contains heap-allocated members (`shared_ptr`, `vector`), so copying `sizeof(QuantileState)` bytes of raw memory does not capture the heap data. On deserialization, `reinterpret_cast`-ing the incomplete bytes back into a `QuantileState` results in dangling pointers and corrupted data.

Additionally, `write_column_to_pb` does not set `PGenericType::QUANTILE_STATE` as the type ID in the `PValues` message.

### What You Expected?

`write_column_to_pb` should use `QuantileState::serialize()` to properly serialize the logical value, and `read_column_from_pb` should use `QuantileState::deserialize()` to reconstruct it. The type ID should be set to `PGenericType::QUANTILE_STATE`.

### How to Reproduce?

Run the BE unit test `QuantileStateSerdeTest.writeColumnToPb` on the unfixed code. The test fails because the round-trip through `write_column_to_pb` / `read_column_from_pb` produces corrupted data:

```
Expected equality of these values:
column_quantile_state->get_data_at(j)
except_column->get_data_at(j)
```

The raw memory comparison shows mismatched bytes because the heap data is lost during serialization.

### Are you willing to submit PR?

Yes, I will submit a PR to fix this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.