internetarchive / internetarchive/openlibrary

Provisioning & Deployment Plan

Open
#4,423 1 comment 3 reactions 0 assignees View on GitHub
Lead: @cdrini Needs: Breakdown Priority: 3 Type: Epic
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

This is a planning epic for our [Provisioning and Deployment project](https://github.com/internetarchive/openlibrary/projects/30) and doesn't require direct attention.
-- To be closed when all underlying issues are resolved.

## Top-level Goals

1. [x] #4060 Deprecate & decommission ol-home in favor of ol-home0
1. [x] Migrate legacy ol-home services (cron jobs, ~solr-updater, import-bot~) to run in Docker on ol-home0
2. [x] ~#4382, #4318, #4383, #4395, #4457~ Automate Deployment (on ol-home0)
3. [x] #4060 Final decomission steps for ol-home
5. [x] #4229 Reprovision dev.openlibrary.org like staging.openlibrary.org (py3)
2. [x] #2036 Re-provision all necessary hosts with `focal` (`ol-www1`, `ol-db[1,2]`, `ol-mem[3-5]`, `ol-backup0`)
3. [ ] #4252 Move all necessary hosts to docker (`ol-www1`, maybe `ol-mem[3-5]`)
4. [ ] #4363 Add jenkins to help automate & schedule deploys, branch staging, and solr rebuilding
5. [ ] #4724 Make our deploys create a detailed GitHub release using https://github.com/github-tools/github-release-notes

## Steps

Many of these still require issues & breakdown

- [x] #4324 Move ImportBot to ol-home0 --> make-a-service - Port ID?, Nagios
- [x] Deploy new dockerized `solr-updater` (to prod)
- [x] #4395 Scripts for production deployments -- Automated (4h -> 15min)
- [x] #4457 Reboot Docker services on specified servers
- [x] internetarchive/olsystem#110 Replace references of ol-home with ol-home0
- [x] #4383 Move cron jobs to ol-home0 (Make one service for all cron jobs -- Caution on the double run)
- [x] #4719 rm volume mounts (for deployments) -- needs issue
- [x] #4252 ol-www1 `epic` (edge cases: logs, ssl-termination, sitemaps) @mekarpeles
- [x] #4252 need to spin up a ol-www0 @mekarpeles
- [ ] deploy strategy needs to address ol-www0, solr updater, import bot, cron service + volume_mounts
- [x] #4060 eventual ol-home deprecation
- [ ] remove ol-home proxypass to ol-home0 for infobase + update config
- [ ] #4426 can we dump stats-solr and move from ol-home to ol-home0
- [ ] can we reclaim 4gb of ram from ol-home (until we can decommission)
- [ ] ol-db[1,2] and ol-backup?

### Stakeholders

@bfalling, @cdrini, @cclauss

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.