Discussion on limiting usage by individual users

## Why open an issue here
While it may be out of scope for dask-gateway to support limiting resource usage for users in a k8s environment, I figured I'd write down an idea I got to solve an issue for dask-gateway users wanting to mitigate the risk of ending up with crazy cloud bills caused by mistakes or mischievious usage.

## Old idea - ResourceQuota (+ k8s native validation admission webhook)
A dask-gateway Helm chart deployment currently can't limit the amount of cloud resources that the individual user consumes, or can it? Assuming it can't, it has been considered if k8s native ResourceQuota resources can help do this, but they can't and are only intended to collectively limit a namespace rather than separate users in a namespace.

## New(?) idea - OurCustomResourceQuota (+ our developed validation admission webhook)
What if we could create a new dedicated open source project that aim to help us mitigate wild usage, while also being generally useful in k8s for other similar projects with pipelines etc created by users? The idea is to create an _validating admission controller_ that would inspect if a pod spec should be approved to be created based on custom logic considering who wish to do so based on pod labels. I imagine we could mimic what the ResourceQuota admission controller does together with the ResourceQuota k8s resources, but let it consider labels in a suitable way and with some new custom k8s resource we define.

If we did this, we could let this resource also monitor the usage by users, which prometheus could collect, and grafana could present for administrators. As a bonus, following this, it wouldn't be too much more effort to let users of a JupyterHub or dask-gateway become more aware of their usage by providing some insights one way or another. Perhaps through a JupyterLab extension, or by injecting information to the page where before a user spawns a server.

## Implementation ideas

- A Helm chart is defined to contain the k8s resources we need
- We need a pod acting as a server listening for request from the k8s-apiserver to allow/deny pod creations
- We need a ValidatingWebhookConfiguration so the k8s-apiserver knows to ask our server for pod creation approval
- We need Role/RoleBinding/ServiceAccount for our server to inspect resources to limit the creation of based on some logic
- We need some TLS stuff to speak with k8s api-server I think.
- Perhaps OPA can be of use to provide general functionality? https://www.openpolicyagent.org/docs/latest/

## References
- [k8s docs - ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/)
- [k8s ref - ResourceQuotas](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#resourcequota-v1-core)
- [k8s docs - Admission controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)
- [k8s ref - Custom admission controllers](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
- [k8s blog post - A guide to admission controllers](https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/)

Ping @rabernat @jhamman @scottyhq @yuvipanda @choldgraf @georgianaelena