Unable to configure TLS for OTEL grpc exporter in code

*Description*:
I'm trying to set up OTEL grpc exporter for my application to push metrics to a gprc endpoint, which requires mTLS secure connection. I want to use `otlpmetricgrpc.WithTLSCredentials()` in [code](https://github.com/envoyproxy/gateway/blob/main/internal/metrics/register.go#L215-L219) to configure TLS certs instead of passing the ENV variable `OTEL_EXPORTER_OTLP_CERTIFICATE` , because it's easier to inject my server's CA cert in code. (This change is made to a local forked version of the gateway).

However it doesn't work. Keep getting the error
```
2024-11-01T23:30:47.853Z INFO provider channelz/logging.go:55 [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "xx.xx.xxx.xxxx:24184", ServerName: "xxx-ingestion-gateway.telemetry.g.xxx.com:24184", }.
Err: connection error: desc = "error reading server preface: remote error: tls: certificate required" {"runner": "provider"}
```
seems like the exporter is not picking up the tls credentials. I've verified the cert files using ENV variable route so those aren't the problem

*Repro steps*:
Code change in [this function](https://github.com/envoyproxy/gateway/blob/main/internal/metrics/register.go#L211)
```
func registerOTELgRPCexporter(otelOpts *[]metric.Option, opts registerOptions) error {
...
tlsConfig, err := crypto.TLSConfigWithCustomCA()
if err != nil {
metricsLogger.Error(err, "failed to create tls config")
}
httpexporter, err := otlpmetricgrpc.New(
context.Background(),
otlpmetricgrpc.WithEndpoint(address),
otlpmetricgrpc.WithTLSCredentials(credentials.NewTLS(tlsConfig)),
)
...
}
```

*Environment*:
latest gateway. Running in Kubernetes.