Create replicas of key systems to enable automatic failover during downtime

### Epic / Tracking Issue for a significant work effort.

I am always frustrated when our service is down for our users.

## Describe the problem that you'd like solved

Our platform is increasingly mission-critical for users around the globe so let's leverage [our Docker-based architecture](https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture#current-production-architecture) to implement automatic failover of key services.

The following list is in the recommended order of implementation:
# Services currently running on multiple servers
* [x] __memcache__ service failover between hosts `ol-mem0`, `ol-mem1`, `ol-mem2` running `memcached` on bare metal
* [x] __database__ failover between hosts `ol-db1` (primary) and `ol-db2` (backup) running Postgres on bare metal
* [x] __web__ service failover between hosts `ol-web1` and `ol-web2` running Docker container `openlibrary-web-1`
* [ ] __Solr__ service failover between hosts `ol-solr0` and `ol-solr1` running Docker container `openlibrary_solr_1`
# Services currently running on a single server
* [ ] __cover images__ service failover between Docker containers `openlibrary-covers-1` and `openlibrary-covers-2`
* [ ] __home__ services running on `ol-home0` running seven different Docker containers
* [ ] __www__ services running on `ol-www0` running `haproxy` and `nginx` Docker containers

It will be important to distinguish services that will operate in primary/backup mode (like `database`) from those which will operate in load-sharing / parallel mode (like `Memcache`). We will need to document and test the failover conditions and constraints. For example, failure of the primary database server might put the site on read-only mode on the backup server.

## Proposal & Constraints

Document and implement a failover approach for each of the services listed above and then use [chaos monkey](https://netflix.github.io/chaosmonkey/)-like testing to ensure service resilience in the face of unplanned software, operating system, and hardware failure.

The hosts in a failover pair must be placed on _different virtual machines_ to ensure resilience to hardware failures. This should also simplify the process of planned downtime and hardware migration while also distributing workloads among virtual machines.

Many of these services might require a two-step migration to failover. The minimum-viable-failover phase will prove basic service failover while documenting but not solving all corner cases. The full failover phase will improve automation and solve all documented corner cases.

## Tracking issue
* [x] Document which of our hosts are on which Kernel-based Virtual Machine hosts and which hosts have SSDs, etc.
* [Current Production Architecture](https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture#current-production-architecture)
* [Internal view](https://github.com/internetarchive/olsystem/wiki/Open-Library-hosts)

__database failover:__
* [ ] Something we should learn as we set up “fail-over” on our servers, is whether Open Library can continue to operate when `ol-db1` goes down (i.e. auto switch to `ol-db2` in read-only mode)
* [ ] Configure `ol-db1` & `ol-db2` networking so that if/when `ol-db1` goes down, Open Library is able to gracefully switch to `ol-db2` in read-only mode
* [ ] `openlibrary.yml` config specifies that `ol-db1` is our database so what change would enable failover?
* [ ] Q: Is it possible (in networking land) for `ol-db2` to take over the IP or hostname of `ol-db1` in the event of an outage?
* [ ] If not, how are these failovers typically done?
* [ ] Q: Should infobase be modified to implement failover when establishing each database session or is there a better approach?
* [ ] Are there significant differences between software v. hardware outages?

__database upgrade:__
* [x] #5686
* [x] #5675

## Stakeholders

@abezella @mekarpeles @cdrini @scottbarnes