Ability to specify which container to inherit ServiceAccount from

Feature Request

What problem are you trying to solve?

We are currently trying to setup a humio cluster in kubernetes using the humio-operator in a centralized logging cluster. The idea is to use linkerd multicluster to create the network mesh between the logging cluster and other clusters. The specific use case in the first place is to only create a mesh between humio in the centralized cluster and fluent-bit running as a DaemonSet in another cluster.

We are currently running into an issue with adding the humio cluster to the mesh. I've created an issue on the humio-operator as well, see https://github.com/humio/humio-operator/issues/204 with steps to reproduce.

The humio-operator uses multiple ServiceAccounts one for each container in the pod. This confuses the linkerd-proxy as it just picks the first VolumeMount:
https://github.com/linkerd/linkerd2/blob/c5d3b281bee75935870b209c75fa28a6f87dcd02/pkg/inject/inject.go#L598

However, in this case it's the wrong one, indicated with this output from the linkerd-proxy:

[    60.70596035s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Failed to certify identity: grpc-status: FailedPrecondition, grpc-message: "requested identity did not match provided token: requested=example-humiocluster-humio.default.serviceaccount.identity.linkerd.cluster.local; found=example-humiocluster-auth.default.serviceaccount.identity.linkerd.cluster.local"

The humio pod contains to containers, auth and humio. Unfortunately, by alphabetic order, the linkerd-proxy gets the auth mount instead of the humio mount. See below:

  linkerd-proxy:
    Container ID:
    Image:          gcr.io/linkerd-io/proxy:stable-2.8.1
    Image ID:
    Ports:          4143/TCP, 4191/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:
         ..... DETAILS OMITTED .....
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from auth-service-account-secret (ro)

How should the problem be solved?

This problem could be solved, if it was possible to instruct the linkerd-proxy to which container to get the mount from by container name or similar.

How would users interact with this feature?

Users would interact with this feature by annotating pods with the container name linkerd should get the VolumeMount from.

Start in pkg/inject/inject.go around the VolumeMount selection referenced at line 598, and review how pod annotations and container names are handled. Define the annotation behavior for selecting the ServiceAccount mount, then add coverage for a pod with multiple container ServiceAccounts and verify the proxy uses the selected mount.