Credential Injector Filter: EOF when retrieving token in oauth_client does not retry

**If you are reporting *any* crash or *any* potential security issue, *do not*
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.**

*Title*: *Credential Injector Filter: EOF when retrieving token in oauth_client does not retry*

*Description*:

Note running on 1.35.1

When the oauth token service that Envoy's credential injector is configured to returns an empty body to a token request this is treated as fatal and the request is never retried.
This type of failure could be transient due to a failure in network components between Envoy and the token service and therefore should be retried.
In general it might be desirable to retry even on receiving malformed tokens in responses since transient errors of this type can occur for example when an intermediate component returns an error message without setting the HTTP response code appropriately.

*Repro steps*:
We are unable to supply repro logs since they contain private information including access tokens. If one were to recreate this manually you would need to arrange for the token server to return a 200 OK http message with an empty body.

Instead of repro data I will provide a test in a PR that demonstrates the problem

>**Note**: The [Envoy_collect tool](https://github.com/envoyproxy/envoy/blob/main/tools/envoy_collect/README.md)
gathers a tarball with debug logs, config and the following admin
endpoints: /stats, /clusters and /server_info. Please note if there are
privacy concerns, sanitize the data prior to sharing the tarball/pasting.

*Admin and Stats Output*:
>Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
[admin endpoint documentation.](https://www.envoyproxy.io/docs/envoy/latest/operations/admin)

I don't believe that the stats output is relevant
in this instance.

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Config*:

```
admin:
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: proxy_server
address:
socket_address:
protocol: tcp
address: 0.0.0.0
port_value: 9090
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: proxy_server
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
route_config:
name: proxy_router
virtual_hosts:
- name: all
domains: ["*"]
routes:
- match:
prefix: "REDACTED"
route:
cluster: "REDACTED"
auto_host_rewrite: true
http_filters:
- name: envoy.filters.http.credential_injector
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.credential_injector.v3.CredentialInjector
overwrite: true
allow_request_without_credential: false
credential:
name: envoy.http.injected_credentials.oauth2
typed_config:
"@type": type.googleapis.com/envoy.extensions.http.injected_credentials.oauth2.v3.OAuth2
scopes:
- REDACTED
token_endpoint:
cluster: oauth
uri: "REDACTED"
timeout: 3s
client_credentials:
client_id: "REDACTED"
client_secret:
name: "oauth-client-secret"
auth_type: URL_ENCODED_BODY
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "REDACTED/tls.crt"
private_key:
filename: "REDACTED/tls.key"
secrets:
- name: "oauth-client-secret"
generic_secret:
secret:
inline_string: "unused"
clusters:
- name: oauth
connect_timeout: 2s
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: oauth
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: "REDACTED"
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: "REDACTED"
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "REDACTED/tls.crt"
private_key:
filename: "REDACTED/tls.key"
- name: REDACTED
connect_timeout: 2s
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: REDACTED
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: "REDACTED"
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: "REDACTED"
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "REDACTED/tls.crt"
private_key:
filename: "REDACTED/tls.key"
```

*Logs*:
>Include the access logs and the Envoy logs.

Logs not included, will provide test.

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Call Stack*:
> If the Envoy binary is crashing, a call stack is **required**.
Please refer to the [Bazel Stack trace documentation](https://github.com/envoyproxy/envoy/tree/main/bazel#stack-trace-symbol-resolution).