Upgrade All Trusty Nodes to Ubuntu Latest

Related to #703 (see aspirational #680)

- [x] Prove provisioning a `generic` minimal xenail VM (e.g. of the `ol-mem` flavor) and add it to the `ol` cluster (e.g. as `ol-mem4`)
- [x] Codify the `ol-mem` approach using Ansible @abezella -- following https://github.com/internetarchive/openlibrary/issues/680#issuecomment-481929471
- [x] Prove provisioning of an `openlibrary-specific` xenial VM (e.g. `ol-web3`) using Docker and add it to the `ol` cluster (e.g. as `ol-web1`). This requires preserving the existing `/opt/openlibrary` directory as a detachable volume and preserving to new instance.
- [x] Codify the `openlibrary-specific` approach (using Ansible + Docker) such that new `ol-web` xenial nodes can be added automatically into the `ol` pool.

[View Architecture & Provisioning docs on the Wiki](https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture)

## Remaining Trusty Machines

- [x] ol-www1.us.archive.org → #4252
- [ ] ol-home.us.archive.org → #5143
- [x] #4775
- [x] #5144
- [x] #7817
- [ ] ol-db1.us.archive.org → #5686
- [ ] ol-db2.us.archive.org
- Not sure what is required for ol-db2 in terms of migrating replication & provisioning as I don't think we have any record of how this vm was built
- [ ] ol-backup0.us.archive.org →
- We don't have much context on ol-backup0 and what is required.
- Is it possible for us to just spin up an ol-backup1 and temporarily write backups to **both** locations and verify that's working and then simply de-comission ol-backup0?
## Requirements
The Trusty 14.04 release of the Ubuntu operating system will reach end of life for LTS (long-term support) at the end of 2019. After this time, our VMs may no longer receive necessary security updates. Therefore, before 2020, we are required to re-provision all our ~11 production Open Library VMs to run Xenail.

## Current Production Architecture
Today, our production service architecture consists of ~11 VMs:
(see: https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture)

## Current Provisioning Setup
Our current production setup process (as of 2019) for provisioning these 11 VMs is ostensibly **manual** and relies on a lot of manually `scp`ing directories around, as well as a separate repository called `olsystem` which contains the production configs, cron jobs, and infrastructure required to run the official openlibrary.org service.

Each of our 11 VMs are more-or-less provisioned identically:
- Every VM has an `/opt` directory containing all the "business"
- Within `/opt` there is an `openlibrary/` and a `petabox/` directory. It's very likely `/opt/petabox` is not required by all VMs, though it's not currently well understood which services may rely on it (e.g. the `ol-home` VM makes heavy use of `olsystem` which may reference petabox)
- `/opt/openlibrary` contains all the business logic for the Open Library project:

```
/opt/
/opt/petabox
/opt/openlibrary
/opt/openlibrary/venv -- python virtualenv
/opt/openlibrary/maxmind-geoip/ -- .dat file for anonymizing IPs
/opt/openlibrary/deploys -- history of all deploys, hash-binned by service
/opt/openlibrary/deploys/openlibrary -- history of openlibrary deploys
/opt/openlibrary/deploys/olsystem -- history of openlibrary deploys
/opt/openlibrary/deploys/base -- deprecated??
/opt/openlibrary/deploys/openlibrary/openlibrary -- active openlibrary deploy
/opt/openlibrary/deploys/openlibrary/openlibrary -- active olsystem deploy
/opt/openlibrary/olsystem/ -- symlink to active olsystem: /opt/deploys/openlibrary/olsystem
/opt/openlibrary/openlibrary -- symlink to active openlibrary: /opt/deploys/openlibrary/olsystem
```

## Minimum Proposal
At minimum, re-provisioning a VM requires:
- setting up firewall rules and installing core packages (e.g. git, docker) by running an ansible playbook
- `scp`'ing over the legacy VM's `/opt` directory (preferably as an external mountable `/1` volume which can be moved in the future)
- Setting up `olsystem` so that its files within `/opt/openlibrary/olsystem/etc` symlink to the right locations within `/etc`

To copy `/opt` over from another server you'll have to:
on `ol-mem2`: `sudo tar cpSlf /var/tmp/ol.tar --same-owner -C /opt openlibrary`
`scp /var/tmp/ol.tar ol-mem4:/var/tmp/ol.tar`
on `ol-mem4`: `tar xpBsf /var/tmp/ol.tar --same-owner -C /opt` (edited)
(due to keys and needing to be root to get all of it i don't there's an easy way to just scp or rsync)

## Ideal Proposal
An aspirational goal of this epic is to migrate Open Library VM provisioning to use a standard Ansible playbook (and possibly docker containers, a la our development environment) to support this re-provisioning.

Part of this effort includes decreasing production's dependence on the `olsystem` repository a la #680. Both developer and production systems should use have similar docker recipes and differ according to ansible playbooks.

## Plan
The plan is to start with `ol-mem0`, `ol-mem1`, and `ol-mem2` as they don't really require any infrastructure other than:
1) setup 3 new memcached servers `ol-mem3`, `ol-mem4`, `ol-mem4`
2) provision VMs with default ansible playbook: setup firewall rules + install docker
3) use VM-specific ansible playbook to install setup docker w/ memcached (with upstart)
4) update /opt/openlibrary/olsystem/etc/openlibrary.yml and infobase.yml configs to reference correct new memcached servers
5) /etc (e.g. memcached) to symlink to the correct system configs in `/opt/openlibrary/olsystem/etc/`
6) update /opt/openlibrary/olsystem/fabfile.py supervisord to update how memcached servers should be restarted (and to not deploy to ol-mem* during deploy)
7) remove old memcached servers from the pool (one at a time)