Dealing with containers in ContainerCreating state

The operator seems to have a problem with pods which are stuck in the state ContainerCreating. Pods might be stuck there, for example, because a volume cannot be mounted.

The problem has an impact on the operator itself because it blocks the workers with processes starting to recreate pods. In this case no more Postgres clusters can be created because the events are stuck in the queue.

Operator v1.3.1

http://localhost:8080/workers/all/status

{
  "0": {
    "CurrentCluster": {
      "Namespace": "k8s-pgbouncer-sidear-test-0p560nsg",
      "Name": "improvement-test-cluster"
    },
    "CurrentProcess": {
      "Name": "starting to recreate pods",
      "StartTime": "2020-01-31T14:49:38.352382396Z"
    }
  },
  "1": {
    "CurrentCluster": {
      "Namespace": "postgres-sidecar-test",
      "Name": "improvement-test-cluster"
    },
    "CurrentProcess": {
      "Name": "starting to recreate pods",
      "StartTime": "2020-01-31T14:49:01.419016514Z"
    }
  },
  "2": {
    "CurrentCluster": {
      "Namespace": "k8s-pgbouncer-sidear-test-ukga5av8",
      "Name": "improvement-test-cluster"
    },
    "CurrentProcess": {
      "Name": "starting to recreate pods",
      "StartTime": "2020-01-31T14:47:54.455182054Z"
    }
  },
  "3": {
    "CurrentCluster": {
      "Namespace": "bfischer-sandbox",
      "Name": "improvement-test-cluster"
    },
    "CurrentProcess": {
      "Name": "starting to recreate pods",
      "StartTime": "2020-01-31T14:48:16.282425935Z"
    }
  }
}

Start with the /workers/all/status endpoint and the reported starting to recreate pods process, then examine behavior when a pod remains in ContainerCreating because a volume cannot be mounted. Done should include preventing the blocked workers from stalling the event queue and allowing other PostgreSQL clusters to be created.