GSA / GSA/data.gov

Harvester DB Migration Rebuild

Open
#6,258 0 comments 0 reactions 0 assignees View on GitHub
Harvest-DB
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

## User Story

In order to release DB migrations intentionally and with purpose, data.gov admins want DB migration split into a separate process and completed when PR has tag (or when PR has migrations in it).

## Acceptance Criteria

- [ ] GIVEN a DB migration is prepped and ready to be applied to staging/production \
WHEN the migration changes are accepted and merged into main \
THEN the migration occurs smoothly on release \
AND if any failures occur the change is rolled back and issues are created to manage/mitigate \
AND deploys are stopped until triage occurs

## Background

See https://github.com/GSA/data.gov/issues/6147 and https://github.com/GSA/data.gov/issues/6145. We have schema drift that wasn't being reported and in a bad state.
We want to streamline the process to be better.
We did this for complex OpenSearch changes/migrations, which was more complex because it involved a whole new index and coordinating with the catalog application. See https://github.com/GSA/datagov-harvester/pull/842 and https://github.com/GSA/data.gov/issues/5885

## Security Considerations ([required](https://nvd.nist.gov/800-53/Rev4/control/CM-4))

[comment]: # "Our SSP says 'The Data.gov team ensures security implications are considered as part of the agile requirements refinement process by including a section in the issue template used as a basis for new work.' so please don't remove this section without care."
None

## Sketch

Current migration happens as part of startup: https://github.com/GSA/datagov-harvester/blob/main/app-start.sh#L9

This is going to be a blue/green deployment. What that means is we have 2 versions of the application running in parallel in order to service the in-between state of the DB (partial migration has occurred).

The workflow should be something like the following:

- Stop all new harvest jobs, wait/kill current running jobs (wait 10 minutes, then kill; can always be re-run)
- Create new harvest application with new code and deploy, scaled to 0, with 0 harvest runners.
- Start task specifically for DB migration
- Once task is confirmed running, scale new app to 1
- Wait/watch for DB migration task to finish and succeed
- Scale old app to 0, new app to 2, and swap names (datagov-harvest becomes datagov-harvest-old and datagov-harvest-new becomes datagov-harvest)
- Verify everything working and all good
- Resume harvest job runners and remove old app
- If at any point any of the above fails, create ticket and disable deployment action (and possibly other O&M notification, ie slack?)

This is meant to provide the best user experience that if the DB is in an in-between state the new application could serve the request and be successful, and vice-versa. It will be buggy, but the only "sure fire" way to keep everything alive would be to have scheduled down time (turn off the app and run the migration and turn it back on with new code); which is possible but seems unnecessary. We could also consider not turning on the blue/green deployment and letting certain routes/items fail until able to spin up the new application after the migration is complete.

Contributor guide

Open the contributing guide

Research direction

Start with app-start.sh at line 9, then review issues 6147 and 6145 and the referenced OpenSearch migration work in datagov-harvester PR 842. Done means implementing and validating the documented blue/green migration workflow, including failure rollback, deploy blocking, and resuming harvest jobs after verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
databases, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.