internetarchive / internetarchive/openlibrary

Followup to Psql 18 Migration

Open
#12,199 3 comments 0 reactions 0 assignees View on GitHub
Lead: @mekarpeles Needs: Breakdown Needs: Staff / Internal Priority: 2 Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Feature Request

## Problem / Opportunity

https://github.com/internetarchive/openlibrary/issues/5675#issuecomment-4136966418

## Next Steps

1. [x] Add `olsystem` to ol-db0 and consider deploy flow changes for `ol-db0` (up-to-date olsystem)
2. [x] P0 We need to now setup ol-db3 as replica
3. We still have indices we may want to change (e.g. int -> bigint) #10921 #12009
4. We need to now setup ol-db3 as replica
5. We need to update our cron jobs to move backups to *not* ol-backup0??? (which means we need an ol-backup1?)
6. Some plan for decomissioning ol-db1 + ol-db2; cc @abezella
7. ol-db0 **extremely carefully** `sudo rm -rf /2/postgresql/9.3/main` and uninstall 9.3: `sudo apt-get remove postgresql-9.3 postgresql-client-9.3`
8. symlink pg files from olsystem (and ferm as well). NB: the pg files in /2/ contain different path references than /etc and so symlink may not be possible. Either way, ground truth files should be version controlled in olsystem, maybe for both.
9. Update grafana + graphite stats e.g. https://catalogd.archive.org/control/stats/db.php?db=openlibrary#60d

## Proposal

### Breakdown

Implementation Details (for maintainers)

#### Related files

Refer to [this map of common Endpoints](https://docs.openlibrary.org/developers/backend/endpoints.html):
*

#### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]

#### Stakeholders

*


#### Instructions for Contributors

* **Before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked Psql 18 migration discussion and the olsystem configuration for ol-db0 and ol-db3. Then inventory the remaining checklist items covering replication, cron backups, database decommissioning, PostgreSQL file paths, and Grafana/Graphite statistics; completion requires those operational changes to be checked off.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, postgresql
Domain
databases, infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.