In-place retries using ephemeral containers

Following the conclusion of https://github.com/argoproj/argo/issues/2551:

> An update on this: given some limitations placed by K8s on this feature – mainly the inability to replace or modify individual ephemeral containers in a Pod and only replace the entire list of ephemeral containers as an operation – we don't think this feature as described is currently feasible.
>
> However, I'll investigate if we can take advantage of this feature for other purposes, such as a streamlined "Retry" node that performs its retries on the same Pod, saving the need to create new ones and download artifacts every time.

Investigate the potential of using ephemeral containers to do in-place retries within the same pod. When performing in-place retries, containers would be started as ephemeral containers instead of main containers, allowing them to be restarted in-place after they fail. The benefits of this feature would include a more constrained execution context: no need to execute new pods, possibly reaching Node limits, and more streamlined use of artifacts.

---

**Message from the maintainers**:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Start with the Kubernetes ephemeral-container limitations described in this issue and the linked conclusion from issue 2551. Investigate whether retries can run in ephemeral containers within the same Pod, and document a viable design or the blockers to implementation.

Written by the indexing model from the issue text.