Hubs Cloud/HMC big todos

Ops maintenance:

* SSL certificates for janus on HMC need to be managed properly. they expire in october 2020. ideally things will be moved to certbot. however, a wildcard cert could be picked up again instead and packed into the secrets tarball (gpg encrypted) in hubs-ops-secrets, which is where the existing one lives, and the servers rebuilt. the secrets tarball is unpacked by all the servers at boot time. see the terraform/user-data script for the servers.
* domain names need to be kept renewed in the namecheap account set up in the mozilla 1 password vault
* All other domain names and SSL certs are dealt with automatically by AWS. none of this applies to Hubs Cloud

Hubs Cloud major bugs and TODOs:

* The worker stuff needs to be moved from walking people through Cloudflare instead to Stackpath. This has been tested and works, however it costs data transfer (but cheaper than AWS, which is the point.) Cloudflare block risk dictates this change. Ideally Stackpath could be pre-flighted for a while on HMC to verify it works OK with some real traffic. This is one open issue that needs to be figured out when running on stackpath: https://github.com/mozilla/hubs-cloud/issues/56
* There still seems to be issues with SSL certificate creation in Hubs Cloud stack creation - typically its due to DNS not being setup right or AWS quota hit, but we have reports of it not being the only problem.
* Requesting sandbox updates and pre-registering domain names and setting up route 53 has been challenging. These can be automated more as a pre-flight step before going through AWS marketplace. People often request a email quota increase in the wrong region. It seems worth considering tooling out a specific workflow for a 3rd party email provider other than AWS.
* There are a number of issues with digitalocean still: https://github.com/mozilla/hubs-cloud/issues/74 and https://github.com/mozilla/hubs-cloud/issues/68 - also, the original DO workflow assumed sendgrid was a 1-step process. It now requires domain validation. That setup should either be automated in arbortect, or a new email provider recommended that is easier to set up.
* t3.micros have known issues where the site goes down. this is typically due to memory exhaustion. a known repro is deploying a custom client: https://github.com/mozilla/hubs-cloud/issues/65
* There is a known bug in the stack template where domains that end in two-part TLDs do not work due to the domain parser: https://github.com/mozilla/hubs-cloud/issues/75

┆Issue is synchronized with this [Jira Task](https://mozilla-hub.atlassian.net/browse/HUBS-1053)

Start with the terraform/user-data script to understand how the encrypted secrets tarball is unpacked at boot, then review linked issues #56, #74, #68, #65, and #75. This issue needs to be split into separately scoped tasks; completion would require resolving and validating each listed certificate, domain, provider, and infrastructure concern.

Written by the indexing model from the issue text.