Think about whether and how to add support for multiple logs within the same deployment in Terraform

Start by reviewing the existing Terraform module for GCP deployments and the other Terraform modules that add log instances. Separate resources that can be shared by an environment from resources that belong to one log, including the Spanner instance and database, buckets, keys, and service accounts mentioned here. Done should be a decided module structure that supports multiple logs and deleting one log without affecting the environment or other logs.

(I'm talking about this in the context of GCP deployments here, but it probably applies to at least other Cloudy infras and perhaps MySQL too).

Currently, we have a terraform module for creating the services required to deploy a log built with Tessera on GCP, and then other tf modules which build on that to add instances of logs too (e.g. conformance testing).

Given that e.g. a Spanner instance costs money (since you have to spring for at least 100 PU), it would be nice to be able to bin-pack a bunch of logs together and have them share resources. Examples where this may be useful are:

1. a number of low-throughput logs, where each uses << 100 PU for sequencing
2. a time sharded log (e.g. CT) where successive shards might share the spanner instance since load will mostly shift towards subsequent shards over time, but some number of recent-ish shards may remain "open" to writes to collect stragglers.

One way to accomplish this would be to have another tf module, so we'd have:

1. an environment module which creates Tessera-specific resources which can be shared (so Spanner, perhaps service account, KMS, etc.) under some name e.g. prod-eu1
2. an instance module which creates the Tessera-specific per-log resources (e.g., Spanner DB, Bucket, Key, etc.) under some log name (e.g. argon2024) and within a specific environment (e.g. `prod-eu1).

Module (1) would be used manually as a "one-off" operation to create an env, and consumers of Tessera might then have their own terraform config which builds on (2) and deploys their binary configured to point at the appropriate log-specific resources within that env.

Arranging like this would also allow operators to safely turn-down and delete specific logs via terraform, leaving the environment and any other logs it contains untouched.