elastic / elastic/elasticsearch-ruby
"bulk" requests failing due to multiple conflicting Accept headers
- Dominant language
- Ruby
- Stars
- 2k
- Forks
- 615
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug/error/problem**
`bulk` requests from the v9 client to a v9 server can fail with a 400 response caused by a duplicate `Accept` header.
Using tcpdump, I observed these headers being sent:
```
Accept: */*
Content-Type: application/vnd.elasticsearch+x-ndjson; compatible-with=9
accept: application/vnd.elasticsearch+x-ndjson; compatible-with=9
```
And this was the error log from fluentd:
`
[warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2026-08-28 12:21:24 +0000 chunk="65a1a7d518cf9ca154f2901a49baed5b" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({host: \"127.0.0.1\", port: 9200, scheme: \"http\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Incorrect header [Accept]. Only one value should be provided\"},\"suppressed\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept, Content-Type]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=null, Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}}]},\"status\":400}"
`
In my setup, I have fluentd using fluent-plugin-elasticsearch and the elasticsearch client gem to send logs to Elasticsearch. I experienced the problem on version 9.5.0 of the client; downgrading to 8.19.3 solved it.
**Your Environment (please complete the following information):**
- Operating System: RHEL UBI 9
- Ruby Version: 4.0.6 from RHEL AppStream
- Elasticsearch client version: 9.5.0
- Elasticsearch version: 9.5.2
- Any other relevant information:
- used with fluentd 1.19.3 and fluent-plugin-elasticsearch 6.0.0
- faraday 2.14.3
- excon 1.7.0
**Additional context**
I used an AI agent to help me investigate this. It pointed out that #2759 writes the headers as lowercase -- `accept` -- and suggests that the Faraday/Excon HTTP stack isn't smart enough to recognise that, and just adds the uppercase version. I haven't traced that through personally though.
Contributor guide
Research direction
Start by tracing how the v9 client constructs headers for bulk requests and how the Faraday/Excon stack merges them, using the duplicate headers and #2759 as the initial clues. Reproduce the request against Elasticsearch 9.5.2 with the listed client versions; done means a bulk request sends only one compatible Accept header and no longer receives the 400 response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, ruby
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100