Data schema mismatch between merged block
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
We run a realtime table `table1` with fields `X` in **upsert** mode. When a new field `Y` is added to the schema, a simple `select` query in the Pinot explorer will return the following error:
```
[
{
"message": "MergeResponseError:\nData schema mismatch between merged block: [X(DOUBLE)] and block to merge: [X(DOUBLE),Y(DOUBLE)], drop block to merge",
"errorCode": 500
}
]
```
The root cause seems to be that not all records are updated to have the `Y` field after the schema change. In the worst case, some record will never receive an update to contain that `Y` field and the whole table becomes unavailable.
Contributor guide
Research direction
Reproduce the schema change on a realtime upsert table in the Pinot explorer: add field Y after records already contain X, then run a simple select. Trace how blocks with different schemas are merged and verify that records missing Y no longer make the table unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100