Realtime scans from MSQ cannot reliably read complex types
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
When `ScanQueryFrameProcessor` delegates work to a realtime server, it sends a `scan` query and writes the response into a frame. Their serialization when selected in a `scan` query is done by Jackson, and it is common for such objects to have "one-way" serialization (they can be serialized using Jackson, but cannot be deserialized).
For aggregating queries, typically these objects are deserialized by the associated aggregator's `AggregatorFactory#deserialize` method rather than by Jackson. This is problematic for `scan` queries, because they don't use `AggregatorFactory`. This leads to errors when attempting to write the resulting objects to a frame, because they aren't the expected type.
To fix it, we need to serialize and deserialize these complex objects in a coherent way. There are a couple options that come to mind,
- We could add a new parameter to `scan` that causes it to serialize complex objects using `TypeStrategy` rather than allowing Jackson to handle it. MSQ realtime delegation would use this parameter. Then, when reading, we use the same `TypeStrategy` to deserialize the object.
- We could rewrite `scan` queries in MSQ's realtime delegation code to wrap all complex types in `complex_encode_base64`, which would accomplish more or less the same thing- it would use the `TypeStrategy` to serialize the object.
This bug affects all MSQ queries that use realtime data and delegate `scan` involving complex types. An example query would be like:
```sql
SELECT "hllSketch"
FROM tbl
WHERE "foo" = 'bar'
```
Queries that involve aggregations on complex types wouldn't be affected.
Contributor guide
Research direction
Start with ScanQueryFrameProcessor and the MSQ realtime delegation path, then compare scan serialization with AggregatorFactory#deserialize and the relevant TypeStrategy behavior. Use the HLL sketch example query to verify that complex values are serialized and deserialized coherently into the output frame without affecting aggregation queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100