Workflow restriction that each step in DAG needs to be an existing WorkflowTemplate

Summary

Argo Workflow currently supports templateReferencing: Secure which will ensure user can only submit Workflow which uses an existing WorkflowTemplate (and need to use workflowTemplateRef, rather than the deprecated templateRef, w/ quote "You should never reference another template directly on a template object (outside of a steps or dag template)"). Otherwise it will error out with

Error: workflows must use workflowTemplateRef to be executed when the controller is in reference mode

It should also support restriction that user can submit a self-defined DAG but each step in DAG needs to be an existing WorkflowTemplate.

Currently it is not doable, as refer WorkflowTemplate in DAG uses templateRef, and doing so will trigger the same error as above.

Use Cases

• I have an existing set of WorkflowTemplates. I'd like the user to trigger any combination of them by defining their own DAG, but not anything else.
• I have an existing WorflowTemplate. I'd like the user to execute any N-fold of it to run in parallel.

It provide the same security level as workflowTemplateRef, as user can basically curl to run every individual Workflow. But it can fully leverage the visibility of DAG diagram.

Start by tracing the existing templateReferencing: Secure validation and the handling of workflowTemplateRef versus templateRef. Compare the requested DAG behavior with the current restriction and determine the validation and test coverage needed; done means a user-defined DAG can reference only existing WorkflowTemplates while preserving the stated security restriction.