Find a way to check k8s cluster resource before creating a gpu pod

At the moment, the POST /instance API endpoint has no way of knowing what our cluster level GPU capacity is. Whether k8s can actually scale or not. This causes weird issues where the frontend has to just do some guess work by polling the GET /instance API.

It would be nice to figure out a way to deal with this properly.

cc @batpad @ingalls

Start by tracing the POST /instance endpoint and its interaction with Kubernetes GPU capacity, then compare it with the GET /instance polling used by the frontend. Define how the API should determine whether the cluster can scale before creating a GPU pod, and consider the work done when frontend guesswork is no longer required.

Written by the indexing model from the issue text.