Allow complete override of User-Agent Header on a per-RPC level from grpc client libraries

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.20.0

What did you expect to see?

tl;dr

This is a followup from a discussion that was initiated on gitter grpc/grpc channel.
Currently, the grpc/java library reuses the User-Agent from the channel for each RPC
and discards User-Agent by treating it as a reserved header.

However, User-Agent is not a reserved header and this creates complications
when trying to write a proxy-like gRPC service for HTTP endpoints that care
about User-Agent for response customization.

Posting it here to get more ideas about how to resolve this.

quoting the conversation below:

Asad @asadali Jun 04 13:10
What was the underlying reason for the restriction on not allowing User-Agent
to be overridden on a per-call basis? can't seem to find a spec which reserves
the User-Agent string for gRPC/HTTP2 and yet there is code in place in the
libraries (grpc-java/ grpc-go ..) to discard any user-supplied metadata
regarding User-Agent and always use the channel's value eg:
Utils.convertServerHeaders

Asad @asadali Jun 04 13:16
use-case:
client ---> httpSVC-A ---> grpcSVC-B ---> httpSVC-C
how can the client's user-agent be conveyed to httpSVC-C? if A-B have ONLY one
channel open between them with a channel-level User-Agent that can't be
overridden
@ejona86 ^ question regarding user-agent behavior

Eric Anderson @ejona86 Jun 04 14:34
@asadali, user-agent is a built-in feature as gRPC itself sends it. There is
an API to change what gRPC sends, but there didn't seem to be any need to allow
it to be changed per-RPC.

Asad @asadali Jun 04 14:43
@ejona86 we seem to have a use-case in which a per-RPC user-agent will make
things easier for us. The alternate is to use custom metadata fields to
preserve this information. that approach seems non-standard and we were hoping
to avoid it. Will it be possible to include a per-RPC user-agent in gRPC? i
will be happy to code it up. but based on what i read in past issues, this
request was repeatedly turned down.

Eric Anderson @ejona86 Jun 04 14:44
That is a cross-language decision. You would need to make clear what the
use-case for it is. Right now, it isn't clear what the use-case is.
Oh. I see now.
You want to communicate the origin client's user-agent to SVC-C
Yeah. That's not appropriate for user-agent.

Asad @asadali Jun 04 14:45
ack

Eric Anderson @ejona86 Jun 04 14:46
.... unless you are making something closer to a proxy. Maybe. It sort of
seems like a can of worms. It just makes a mess of things.
But I think I understand now.

Asad @asadali Jun 04 14:48
so the intermediate gateways aren't pure proxies but maybe more like
aggregators. in the non-GRPC world, the implementation made an assumption that
User-Agent is the originating client's user-agent. and all intermediate hops
honored that. I agree, that this is a very loose reading of the spec. I feel
the more logical method is to update the user-agent on each hop
however, systems built around that assumption aren't happy when they lose this
info :( IMO, gRPC clients can default to per-channel behavior but the choice
should ultimately be left to the user if they want to override it

Eric Anderson @ejona86 Jun 04 14:51
Well, today the application can't set the entire user-agent. gRPC will always
include itself in the user-agent. I'm trying to check what HTTP says to do for
user-agent and proxies.

Asad @asadali Jun 04 14:52
yeah i can use another opinion on this. and current gRPC behavior is what I am
trying to rationalize. does it need to always include its user-agent?

Eric Anderson @ejona86 Jun 04 17:31
@asadali, proxies do forward the user-agent. We do want to enable grpc proxies,
so that does mean we should forward the user-agent. Although on the server, any
compatibility quirks would generally be with the proxy, not the end-client. So
it still seems muddled, but it does seem we should consider it.