[Planning] Create read-only DB to keep API Highly available

This issue is not yet ready for implementation because it's not fully planned out yet.

Context

Our API goes down every time we deploy. It also becomes unresponsive sometimes if we're processing too hard and causing the DB to get overloaded.

Problem or idea

@davidsmejia has been wanting to add a read-only database. I think this will help with both issues.

We can have the API use the read-only DB for every endpoint other than datasets since that's the only endpoint that isn't read-only.

We can also change our deploy process to deploy the read-only-db in a separate step. This means that our deploy process could:

1. Migrate the existing DB
2. Start a new API that points at the existing DB.
3. Forward traffic to the new API.
4. Migrate the read-only API
5. Restart the old API.
6. Forward traffic back to the old API.
7. Remove the new API.

This process can be broken up such that we can trigger a new type of deploy, api-only, that only does steps 2, 3, 5, 6, 7 and should only be run for deploys that contain no new migrations.

Solution or next step

Kick the tires on this approach some. Will this work? What's the hard part? If this approach seems reasonable then I think this should be broken into at least many sub-issues including but not limited to:

• Create read-only DB
• Make API use read-only DB for as many endpoints as possible.
• Modify deploy process
• Add new deploy tag to trigger api-only deploys.

No files, tests, or entry points are named. Start by mapping the current API, database access, and deployment process, then assess whether a read-only database and separate API-only deploy are feasible. Done means the approach is validated and split into implementation-ready sub-issues with clear migration and traffic-switching requirements.