opensearch-project / opensearch-project/OpenSearch
[BUG] Request-URI Too Large (with large bearer token)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Describe the bug
We use the official OpenSearch docker container and Helm chart to run it in Kubernetes, in combination with Keycloak. Everything works fine with OpenSearch and OpenSearch-Dashboards. But we noticed, that when the Bearer-Token is too large, requests to OpenSearch fail (while Login via Dashboards still work).
Here are the error messages:
% curl -X GET https://opensearch.my.org/_aliases -H "Authorization: Bearer $LARGE_TOKEN"
{"error":{"root_cause":[{"type":"too_long_http_header_exception","reason":"HTTP header is larger than 8192 bytes."}],"type":"too_long_http_header_exception","reason":"HTTP header is larger than 8192 bytes."},"status":400}
% curl -X GET https://os-mngr/_aliases -H "Authorization: Bearer $LARGE_TOKEN"
error: unable to upgrade connection: <html>
<head><title>414 Request-URI Too Large</title></head>
<body>
<center><h1>414 Request-URI Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>
At first we thought it might be an issue with the ingress controller, but the same error occurs when communicating directly with the service.
I couldn't find any issues or discussions about this issue on the internet, the only reference I found was in a comment on ElasticSearch: https://github.com/elastic/elasticsearch/issues/86875#issuecomment-1130793161
We tried increasing the header size, but this did not resolve the issue:
opensearch:
config:
opensearch.yml: |
http.max_header_size: 32kb
....
Am I missing something, or is there a workaround?
Related component
No response
To Reproduce
Generate a large token (in this case it is 20045 bytes long due to a large number of roles) and send a request to OpenSearch.
The relevant Helm chart configucation:
security:
audit:
ignore_users:
- kibanaserver
keycloak:
rolesKey: "groups"
openIdConnectUrl: http://auth.my.org/auth/realms/my-realm/.well-known/openid-configuration"
and the config:
---
_meta:
type: "config"
config_version: 2
config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
authc:
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
clientcert_auth_domain:
description: "Authenticate via SSL client certificates"
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn
challenge: false
authentication_backend:
type: noop
openid_auth_domain:
description: "Authenticate via Keycloak SSO"
http_enabled: true
transport_enabled: false
order: 1
http_authenticator:
type: openid
challenge: false
config:
subject_key: "preferred_username"
roles_key: "{{ .Values.opensearch.security.keycloak.rolesKey }}"
openid_connect_url: "{{ .Values.opensearch.security.keycloak.openIdConnectUrl }}"
openid_connect_idp:
enable_ssl: true
verify_hostnames: false
pemtrustedcas_filepath: "/usr/share/opensearch/config/my-org-cert/my-org-ca.pem"
authentication_backend:
type: noop
authz:
Expected behavior
I would like to be able to increase the allowed header size (of the webserver) to handle larger tokens.
Additional Details
Plugins
- prometheus-exporter
Host/Environment (please complete the following information):
- Chart version: 2.35.0
- OpenSearch version: 2.19.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the provided curl requests against the OpenSearch service, then inspect the Helm chart's opensearch.yml and the HTTP and OpenID authentication configuration. Compare the 8192-byte OpenSearch error with the nginx 414 response; done means identifying which component rejects the header and documenting a confirmed configuration or workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, helm, java, kubernetes, nginx
- Domain
- authentication, backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100