Produce invalid json when template substitution due to special character

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When I was creating a workflow, I encountered an invalid json error.

fail to create workflow with namespace(canal-flow) and name(null), reason: {"code":3,"message":"templates.main cannot finish template replacement because the result was invalid JSON"}

After investigating the codebase, I found that the problem might be occurring during parameter value replacement in executeTemplate.
Currently, the parameter value is directly written into the buffer.
In our case, the parameter value contains special characters. (🏴󠁧󠁢󠁳󠁣󠁴󠁿)

// SubstituteParams returns a new copy of the template with global, pod, and input parameters substituted
func SubstituteParams(tmpl *wfv1.Template, globalParams, localParams Parameters) (*wfv1.Template, error) {
	tmplBytes, err := json.Marshal(tmpl)
	if err != nil {
		return nil, errors.InternalWrapError(err)
	}
	// First replace globals & locals, then replace inputs because globals could be referenced in the inputs
	replaceMap := globalParams.Merge(localParams)
	globalReplacedTmplStr, err := template.Replace(string(tmplBytes), replaceMap, true)
	if err != nil {
		return nil, err
	}
	var globalReplacedTmpl wfv1.Template
	err = json.Unmarshal([]byte(globalReplacedTmplStr), &globalReplacedTmpl)
	if err != nil {
		return nil, errors.InternalWrapError(err)
	}

Reference

func simpleReplace(w io.Writer, tag string, replaceMap map[string]interface{}, allowUnresolved bool) (int, error) {
					...
					replacement = strconv.Quote(replacement)
					replacement = replacement[1 : len(replacement)-1]
					return w.Write([]byte("{{" + nestedTagPrefix + replacement))
					...
}

Reference

However, when the value contains special characters, golang writes a string of special characters in Unicode, making the replaced template json invalid.
e.g. 🏴󠁧󠁢󠁳󠁣󠁴󠁿

{ "name": "test-config", "value": "\U0001f1e7"}

Reproduce in golang playground

---

I think a safer approach would be to use json marshal before writing it to ensure that the value to be replaced is a valid json string.
This approach successfully solves the issue in our own customized argo-workflows.

func simpleReplace(w io.Writer, tag string, replaceMap map[string]interface{}, allowUnresolved bool) (int, error) {
	replacement, ok := replaceMap[strings.TrimSpace(tag)]
	if !ok {
		// Attempt to resolve nested tags, if possible
		if index := strings.LastIndex(tag, "{{"); index > 0 {
			nestedTagPrefix := tag[:index]
			nestedTag := tag[index+2:]
			if replacement, ok := replaceMap[nestedTag]; ok {
				replacement, isStr := replacement.(string)
				if isStr {
					escaped, err :=  json.Marshal(replacement)   <-- here
					if err != nil {
						return 0, err
					}
					replacement = escaped[1 : len(escaped)-1]
					return w.Write([]byte("{{" + nestedTagPrefix + replacement))
				}
			}

Version(s)

v3.6.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: special-char-
spec:
  entrypoint: main
  arguments:
    parameters:
    - name: flag
      value: "🏴󠁧󠁢󠁳󠁣󠁴󠁿"
  templates:
  - name: main
    inputs:
      parameters:
      - name: flag
    container:
      image: alpine:3.19
      command: [sh, -c]
      args:
      - echo "{{inputs.parameters.flag}}"

Logs from the workflow controller

N/A

Logs from in your workflow's wait container

N/A

Start in workflow/common/util.go at SubstituteParams, then inspect util/template/simple_template.go and simpleReplace. Reproduce the issue with the minimal workflow and special-character parameter shown; done means substitution produces valid JSON and the workflow accepts the parameter without the invalid JSON error.