apache / apache/pinot

Allow changing existing Pinot schema for certain scenarios

Open
#7,607 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Currently it seems like Pinot is very rigid about **only** allowing schema changes where a new column has been added. If you need to make any changes to existing columns for e.g changing column from int to long or timestamp format change, which don't affect existing data you are still forced to delete and recreate the table/schema which is very limiting and depending on amount of data that will need to be reprocessed perhaps not even feasible.

Here is a sample scenario, I created a schema where one of the datetimeSpec columns has a ingestion transform
`{
"name": "CreationTimeMillis",
"dataType": "LONG",
"defaultNullValue": 1592703025,
"transformFunction": "fromDateTime(CreateTime, \"yyyy-MM-dd'T'HH:mm:ssZ\")",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}`
now apparently time format specification here is incorrect and I need to change this to

`{
"name": "CreationTimeInMillis",
"dataType": "LONG",
"defaultNullValue": 1592703058,
"transformFunction": "fromDateTime(CreateTime, 'yyyy-MM-dd''T''HH:mm:ssZ')",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}`
but this is now considered as a backward incompatible change.

Can we add some additional intelligence in the logic so such changes which don't affect underlying data are allowed to go through ?

cc @mayankshriv

Contributor guide

Open the contributing guide

Research direction

No file, test, or specific entry point is named in the issue. Start by tracing Pinot's schema-change compatibility logic for existing columns, then determine how it distinguishes changes that do not affect stored data; done means those compatible type or timestamp-format changes are accepted while incompatible changes remain rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.