ephemeralrunner should support a TTL

What would you like added?

It would be nice if the ephemeral runners supported a TimeToLive value to cleanly expire the pod & ephemeral runners.

Why is this needed?

I have an ephemeralrunner setup defined via gha-runner-scale-set. I have minRunners set, and when the runners startup they connect to GHE and wait for work. Part of my pod spec includes a custom volume that I need to "keep fresh" by rotating every 2 hours. (The volume is time consuming to setup, and our volume provisioner abstracts that time). We installed a liveness probe on the pod to expire after 2 hours, and this works up to a max of 5 times due to this code:

https://github.com/actions/actions-runner-controller/blob/1dbb88cb9ee8fcfc4c28406e5d0b910d8a4419cc/controllers/actions.github.com/ephemeralrunner_controller.go#L190-L198

After hitting that condition, the ephemeralrunner resource hangs around with no new pod and eventually (after ~10 hours of idle time) all ephemeralrunners have no running pods (meaning we lose all our GH Runners).

Start in controllers/actions.github.com/ephemeralrunner_controller.go around lines 190-198, then trace how EphemeralRunner resources and pods are created and cleaned up. Define how a TimeToLive value should expire both the pod and ephemeral runner without exhausting the existing retry condition; done means the resource is replaced or removed cleanly after the TTL and runners remain available.