apache / apache/pinot

Data schema mismatch between merged block

Open
#7,694 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We run a realtime table `table1` with fields `X` in **upsert** mode. When a new field `Y` is added to the schema, a simple `select` query in the Pinot explorer will return the following error:

```
[
{
"message": "MergeResponseError:\nData schema mismatch between merged block: [X(DOUBLE)] and block to merge: [X(DOUBLE),Y(DOUBLE)], drop block to merge",
"errorCode": 500
}
]
```

The root cause seems to be that not all records are updated to have the `Y` field after the schema change. In the worst case, some record will never receive an update to contain that `Y` field and the whole table becomes unavailable.

Contributor guide

Open the contributing guide

Research direction

Reproduce the schema change on a realtime upsert table in the Pinot explorer: add field Y after records already contain X, then run a simple select. Trace how blocks with different schemas are merged and verify that records missing Y no longer make the table unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.