Faster, cheaper and more robust workflow with Emissary v2

The Emissary is a bit like I'm calling the "commandlet pattern". We don't need the wait container with the Emissary. It is in-process, it can do everything wait can do easier.

Basically, the Emissary loads and save artifacts, not the init or wait container. No need to do complex volume mounting, or copying files around. Cheaper and faster.

Pros:

Generally, we benefit of being tightly coupled with the process:

* Obviates the needs for any executor code. Remove all the code and reduce maintenance costs.
* No need to copy files between containers using tricks.
* Cheaper: you use the same resource the main container does.
* Correct signal handling always, comandlet fulfils init role.
* Each "main" container gets its own sidecar, may open up now opportunities.
* Simpler start-up and shutdown sequence (just SIGTERM).
* You only need to set resources on main container. Could reduce overall cost.

Cons:

Generally, we lose the benefit of de-coupling:

* Two processes in on container is odd, might cause un-expected problems.
* Removes the firebreak between the sidecar and the main container. Edgecase issues become harder to debug.
* Port number conflicts.
* Intermingles logs, which is confusing.
* Could be more expensive if sidecar need bigger resources than main container.

We can prove this concept in dataflow first.

---

**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

This is a broad architectural proposal to replace the wait container with an in-process Emissary sidecar workflow. Start by reviewing the dataflow implementation and its current artifact-loading and saving path; done would require a working proof of concept plus evidence that startup, shutdown, resource use, signals, logging, and container interaction meet the stated goals.

Written by the indexing model from the issue text.