Fixing devtime serialization errors for incompatible db schemas -- Part 2
Nobody has claimed this yet.
- Dominant language
- Clojure
- Stars
- 49.3k
- Forks
- 6.8k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 653
Description
# Part 2 - Fixing devtime serialization errors for incompatible db schemas
To reiterate the problem at a high level:
Some changes to the schema can break serializations. Serialization was not
designed to be compatible between versions.
For the production system, the schema changes are not a problem because we will
have an up-to-date dump of instance-analytics content, and a dump+load from the
same (or a changed, but compatible) schema will always work.
The problem is that during development we make changes to the shcema that break
loading new versions of the instance analytics content.
## What breaks it?
1. Deleting a _required_ column
2. Adding a _required_, _defaultless_ column
### Other operations are combinations of those situations:
- Renaming a column: that's deleting the column, and adding a new one. it won't
break anything if neither get required.
- Changing a column's type: that's deleting the column and adding a new one with
an incompatible type. If it has a default, or if the types are close enough
(e.g. varchar(20) -> varchar(100)) that will be OK.
## What fixes it?
### For Deleting a _required_ column
We have a PR to skip values not in the target table on the loading Metabase
instance.
### Adding a _required_, _defaultless_ column
**This issue aims to fix this piece.**
# Approach
This problem is tricky, because IA content failing to load will be a blocker for
CI, so we need to offer a workaround.
1. Detect when a load is going to be impossible.
2. Log an error explaining how to run the *script*.
3. Running the script can generate an updated version of the IA content.
## What does the script do?
1. Roll back the db schema to a time you can load it
2. Load the IA content
3. Up Migrate
4. Dump the IA content with the new version of the schema
5. commit the changes, along with adding the migration
6. now, when restarting the app, we can load IA content on the newest version
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating the instance-analytics serialization/load path and schema migration tooling, then trace how incompatible required defaultless columns are detected. Done means the documented script can regenerate compatible IA content after rolling back, loading, migrating, and dumping the schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure
- Domain
- database
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100