Harvester DB Migration Rebuild
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
## User Story
In order to release DB migrations intentionally and with purpose, data.gov admins want DB migration split into a separate process and completed when PR has tag (or when PR has migrations in it).
## Acceptance Criteria
- [ ] GIVEN a DB migration is prepped and ready to be applied to staging/production \
WHEN the migration changes are accepted and merged into main \
THEN the migration occurs smoothly on release \
AND if any failures occur the change is rolled back and issues are created to manage/mitigate \
AND deploys are stopped until triage occurs
## Background
See https://github.com/GSA/data.gov/issues/6147 and https://github.com/GSA/data.gov/issues/6145. We have schema drift that wasn't being reported and in a bad state.
We want to streamline the process to be better.
We did this for complex OpenSearch changes/migrations, which was more complex because it involved a whole new index and coordinating with the catalog application. See https://github.com/GSA/datagov-harvester/pull/842 and https://github.com/GSA/data.gov/issues/5885
## Security Considerations ([required](https://nvd.nist.gov/800-53/Rev4/control/CM-4))
[comment]: # "Our SSP says 'The Data.gov team ensures security implications are considered as part of the agile requirements refinement process by including a section in the issue template used as a basis for new work.' so please don't remove this section without care."
None
## Sketch
Current migration happens as part of startup: https://github.com/GSA/datagov-harvester/blob/main/app-start.sh#L9
This is going to be a blue/green deployment. What that means is we have 2 versions of the application running in parallel in order to service the in-between state of the DB (partial migration has occurred).
The workflow should be something like the following:
- Stop all new harvest jobs, wait/kill current running jobs (wait 10 minutes, then kill; can always be re-run)
- Create new harvest application with new code and deploy, scaled to 0, with 0 harvest runners.
- Start task specifically for DB migration
- Once task is confirmed running, scale new app to 1
- Wait/watch for DB migration task to finish and succeed
- Scale old app to 0, new app to 2, and swap names (datagov-harvest becomes datagov-harvest-old and datagov-harvest-new becomes datagov-harvest)
- Verify everything working and all good
- Resume harvest job runners and remove old app
- If at any point any of the above fails, create ticket and disable deployment action (and possibly other O&M notification, ie slack?)
This is meant to provide the best user experience that if the DB is in an in-between state the new application could serve the request and be successful, and vice-versa. It will be buggy, but the only "sure fire" way to keep everything alive would be to have scheduled down time (turn off the app and run the migration and turn it back on with new code); which is possible but seems unnecessary. We could also consider not turning on the blue/green deployment and letting certain routes/items fail until able to spin up the new application after the migration is complete.
Contributor guide
Research direction
Start with app-start.sh at line 9, then review issues 6147 and 6145 and the referenced OpenSearch migration work in datagov-harvester PR 842. Done means implementing and validating the documented blue/green migration workflow, including failure rollback, deploy blocking, and resuming harvest jobs after verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- databases, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100