Design: How should substrate use (or not use) namespaces in a Kubernetes cluster?

## Current state:

Not a ton of thought has gone into the current design.

### ActorTemplates

CRDs are cluster-global, but the CRs they describe can be namespaced. `ActorTemplate` is a namespaced CR. The substrate code today pays attention to all `ActorTemplate`s, regardless of namespace. Most of the UX involves specifying templates as a tuple of {namespace, name}. For example: `kubectl ate create actor foo --template bar/my-template` creates an actor named "foo" from the `ActorTemplate` named "my-template" in k8s namespace "bar".

### WorkerPools

Like `ActorTemplate`, `WorkerPool` is a namespaced CRD which is ultimately manifested as k8s pods in that same namespace.

### Control plane

To run Agent Substrate, we need a namespace in which to run a bunch of stuff, such as the control plane.

## How should substrate use k8s namespaces?

Kubernetes namespaces are generally used for separating teams or “tenants”, environments (staging vs. prod), service producers, for managing quota and charge-backs, and many other reasons.

So now we need to answer a fundamental question - is a substrate a cluster level concept (i.e. any cluster can have at most 1 substrate) or can a single cluster run multiple instances of substrate?

It **seems** obvious (to me) that a single cluster should be able to run multiple instances of the substrate. So how would one deploy two instances of it in a cluster? Obviously, those would go into different namespaces, but exactly how to map those is not obvious.

"Workers" are analogous to Kubernetes Nodes. Kubernetes Nodes are not namespaced, which means that Pods can share a Node even if the Pods are from different Namespaces. From that point of view it seems natural that Workers shouldn't be namespace, either. Especially if we want cluster-wide worker-pools that are fungible across actors. But do we?

If a cluster can be "multi-tenant" (with varying levels of hard/soft isolation possible), then a cluster-wide ANYTHING will be trouble. In fact, it's probably not controversial to say that many of the “cluster scoped” (non-namespaced) concepts in Kubernetes are considered problematic in hindsight.

On the other hand, Kubernetes Pods are namespaced, and Workers are implemented as Pods. Making Workers non-namespaced while still semantically linked to a Pod would require complex extra work, and will forever be a vector for security issues. We should not make the presumed multi-tenancy requirements worse. Instead, all of the Workers for a given substrate instance can be k8s Pods within that substrate’s k8s Namespace(s), and those Workers would only service that single instance of substrate.

As shown above, substrate already uses multiple namespaces, and we probably don't want to put things like ActorTemplates (which are part of using substrate) in a namespace with the control plane. It's likely that different personas will use those. So every instance of substrate needs 2 namespaces. At least.

We should probably support a model which uses multiple namespaces for ActorTemplates, and possible for Workers (e.g. to segregate different groups of Workers from each other). Sp really substrate needs a way to say:
* My control plane is in namespace A
* My workerpools are in namespaces B, C, D...
* My ActorTemplates are in namespaces E, F, G...

The usual way to do N-ary things in kubernetes is label selectors.

This model naturally aligns to many of the other primitives we've got in Kubernetes such as workload autoscaling, Gateway and HTTP routes, etc. Some things like network policies can be made to work across namespaces. Some things like quota become more difficult.