envoyproxy / envoyproxy/gateway

An extremely large number of requests to jwt.providers.remoteJWKS.uri

Open
#9,378 4 comments 0 reactions 0 assignees View on GitHub
stale triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:
An extremely large number of requests to jwt.providers.remoteJWKS.uri

I have approximately five SecurityPolicy resources that use Keycloak. An example of the default SecurityPolicy is below.

Periodically, around 2,000 requests are sent within five minutes to the following URL:

https://keycloak.test.net/realms/test/protocol/openid-connect/certs

```
┌───────────timestamp─┬─path───────────────────────────────────────────┬─response_code─┬─requests_count─┐
1. │ 2026-06-30 08:45:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 300 │
2. │ 2026-06-30 08:55:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 7 │
3. │ 2026-06-30 09:00:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 13 │
4. │ 2026-06-30 09:20:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 116 │
5. │ 2026-06-30 09:20:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 2786 │
6. │ 2026-06-30 09:25:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 50 │
7. │ 2026-06-30 09:35:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 32 │
8. │ 2026-06-30 09:55:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 40 │
9. │ 2026-06-30 10:00:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 88 │
10. │ 2026-06-30 10:05:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 604 │
11. │ 2026-06-30 10:10:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 180 │
12. │ 2026-06-30 10:15:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 10 │
13. │ 2026-06-30 10:15:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 642 │
14. │ 2026-06-30 10:20:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 90 │
15. │ 2026-06-30 10:30:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 460 │
16. │ 2026-06-30 10:35:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 613 │
17. │ 2026-06-30 10:40:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 4 │
18. │ 2026-06-30 10:40:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 777 │
19. │ 2026-06-30 10:45:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 38 │
20. │ 2026-06-30 10:50:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 2 │
21. │ 2026-06-30 10:50:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 595 │
22. │ 2026-06-30 10:55:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 18 │
23. │ 2026-06-30 11:00:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 319 │
24. │ 2026-06-30 11:05:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 8 │
25. │ 2026-06-30 11:10:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 8 │
26. │ 2026-06-30 11:15:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 3 │
27. │ 2026-06-30 11:15:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 791 │
28. │ 2026-06-30 11:20:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 3 │
29. │ 2026-06-30 11:20:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 450 │
30. │ 2026-06-30 11:25:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 18 │
31. │ 2026-06-30 11:25:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 574 │
32. │ 2026-06-30 11:30:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 28 │
33. │ 2026-06-30 11:30:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 2945 │
34. │ 2026-06-30 11:35:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 68 │
35. │ 2026-06-30 11:35:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 2609 │
36. │ 2026-06-30 11:40:00 │ /realms/test/protocol/openid-connect/certs │ 0 │ 17 │
37. │ 2026-06-30 11:40:00 │ /realms/test/protocol/openid-connect/certs │ 200 │ 1180 │
└─────────────────────┴────────────────────────────────────────────────┴───────────────┴────────────────┘
```

I would like to understand why around 2,000 requests are being sent and whether there is any way to reduce the number of requests to Keycloak. In my opinion, this is clearly excessive.

*Repro steps*:
```
kind: SecurityPolicy
metadata:
name: alertmanager
namespace: monitoring
spec:
authorization:
defaultAction: Deny
rules:
- action: Allow
name: allow-internal-cidrs
principal:
clientCIDRs:
- 10.10.252.0/22
jwt:
claims:
- name: groups
valueType: StringArray
values:
- /alertmanager.test.net
provider: keycloak
jwt:
providers:
- claimToHeaders:
- claim: email
header: X-Auth-Request-Email
- claim: groups
header: X-Auth-Request-Groups
- claim: preferred_username
header: X-Auth-Request-Preferred-Username
- claim: name
header: X-Auth-Request-User
issuer: https://keycloak.test.net/realms/test
name: keycloak
remoteJWKS:
backendSettings:
retry:
numRetries: 4
perRetry:
backOff:
baseInterval: 100ms
maxInterval: 1s
retryOn:
triggers:
- gateway-error
- reset
- reset-before-request
- connect-failure
- retriable-4xx
- refused-stream
- retriable-status-codes
- cancelled
- deadline-exceeded
- internal
- resource-exhausted
- unavailable
timeout:
http:
connectionIdleTimeout: 4s
requestTimeout: 20s
tcp:
connectTimeout: 150ms
cacheDuration: 4h
uri: https://keycloak.test.net/realms/test/protocol/openid-connect/certs
oidc:
clientID: alertmanager.test.net
clientSecret:
group: ""
kind: Secret
name: alertmanager-keycloak-basic-auth-client-secret
cookieNames:
accessToken: AccessToken-alertmanager
idToken: IdToken-alertmanager
forwardAccessToken: true
passThroughAuthHeader: true
provider:
backendSettings:
retry:
numRetries: 5
perRetry:
backOff:
baseInterval: 100ms
maxInterval: 1s
retryOn:
triggers:
- gateway-error
- reset
- reset-before-request
- connect-failure
- retriable-4xx
- refused-stream
- retriable-status-codes
- cancelled
- deadline-exceeded
- internal
- resource-exhausted
- unavailable
timeout:
http:
connectionIdleTimeout: 4s
requestTimeout: 30s
tcp:
connectTimeout: 150ms
issuer: https://keycloak.test.net/realms/test
redirectURL: https://alertmanager.test.net/login
refreshToken: true
scopes:
- groups
- email
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: alertmanager
status:
ancestors:
- ancestorRef:
group: gateway.networking.k8s.io
kind: Gateway
name: envoy-gateway-internal
namespace: envoy-gateway-internal
conditions:
- lastTransitionTime: "2026-06-30T11:37:50Z"
message: Policy has been accepted.
observedGeneration: 18
reason: Accepted
status: "True"
type: Accepted
controllerName: gateway.envoyproxy.io/envoy-gateway-internal-gatewayclass-controller
```

*Environment*:
Helm chart v1.8.1
envoyproxy/envoy: distroless-v1.38.1
envoyproxy/gateway: v1.8.1
envoyproxy/ratelimit: ff287602

*Logs*:
```
Body: [2026-06-30 11:32:07.482][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:117] Jwks async fetching url=https://keycloak.test.net/realms/test/protocol/openid-connect/certs: failed
Body: [2026-06-30 10:36:08.304][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:117] Jwks async fetching url=https://keycloak.test.net/realms/test/protocol/openid-connect/certs: failed
Body: [2026-06-30 10:36:08.304][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:117] Jwks async fetching url=https://keycloak.test.net/realms/test/protocol/openid-connect/certs: failed
Body: [2026-06-30 10:36:08.304][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:117] Jwks async fetching url=https://keycloak.test.net/realms/test/protocol/openid-connect/certs: failed
Body: [2026-06-30 10:36:08.304][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:117] Jwks async fetching url=https://keycloak.test.net/realms/test/protocol/openid-connect/certs: failed
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the SecurityPolicy configuration with remoteJWKS cacheDuration set to 4h, then inspect the Envoy JWT fetch behavior and the jwks_async_fetcher.cc:117 warnings shown in the logs. Trace why failed fetches generate repeated requests and verify that the completed change explains the request volume and reduces or prevents unnecessary Keycloak requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
authentication, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.