Allow changing existing Pinot schema for certain scenarios
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Currently it seems like Pinot is very rigid about **only** allowing schema changes where a new column has been added. If you need to make any changes to existing columns for e.g changing column from int to long or timestamp format change, which don't affect existing data you are still forced to delete and recreate the table/schema which is very limiting and depending on amount of data that will need to be reprocessed perhaps not even feasible.
Here is a sample scenario, I created a schema where one of the datetimeSpec columns has a ingestion transform
`{
"name": "CreationTimeMillis",
"dataType": "LONG",
"defaultNullValue": 1592703025,
"transformFunction": "fromDateTime(CreateTime, \"yyyy-MM-dd'T'HH:mm:ssZ\")",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}`
now apparently time format specification here is incorrect and I need to change this to
`{
"name": "CreationTimeInMillis",
"dataType": "LONG",
"defaultNullValue": 1592703058,
"transformFunction": "fromDateTime(CreateTime, 'yyyy-MM-dd''T''HH:mm:ssZ')",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}`
but this is now considered as a backward incompatible change.
Can we add some additional intelligence in the logic so such changes which don't affect underlying data are allowed to go through ?
cc @mayankshriv
Contributor guide
Research direction
No file, test, or specific entry point is named in the issue. Start by tracing Pinot's schema-change compatibility logic for existing columns, then determine how it distinguishes changes that do not affect stored data; done means those compatible type or timestamp-format changes are accepted while incompatible changes remain rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100