elastic / elastic/elasticsearch-ruby

"bulk" requests failing due to multiple conflicting Accept headers

Open
#3,059 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
2k
Forks
615
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug/error/problem**
`bulk` requests from the v9 client to a v9 server can fail with a 400 response caused by a duplicate `Accept` header.

Using tcpdump, I observed these headers being sent:

```
Accept: */*
Content-Type: application/vnd.elasticsearch+x-ndjson; compatible-with=9
accept: application/vnd.elasticsearch+x-ndjson; compatible-with=9
```

And this was the error log from fluentd:

`
[warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2026-08-28 12:21:24 +0000 chunk="65a1a7d518cf9ca154f2901a49baed5b" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({host: \"127.0.0.1\", port: 9200, scheme: \"http\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Incorrect header [Accept]. Only one value should be provided\"},\"suppressed\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept, Content-Type]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=null, Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}}]},\"status\":400}"
`

In my setup, I have fluentd using fluent-plugin-elasticsearch and the elasticsearch client gem to send logs to Elasticsearch. I experienced the problem on version 9.5.0 of the client; downgrading to 8.19.3 solved it.

**Your Environment (please complete the following information):**
- Operating System: RHEL UBI 9
- Ruby Version: 4.0.6 from RHEL AppStream
- Elasticsearch client version: 9.5.0
- Elasticsearch version: 9.5.2
- Any other relevant information:
- used with fluentd 1.19.3 and fluent-plugin-elasticsearch 6.0.0
- faraday 2.14.3
- excon 1.7.0

**Additional context**
I used an AI agent to help me investigate this. It pointed out that #2759 writes the headers as lowercase -- `accept` -- and suggests that the Faraday/Excon HTTP stack isn't smart enough to recognise that, and just adds the uppercase version. I haven't traced that through personally though.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the v9 client constructs headers for bulk requests and how the Faraday/Excon stack merges them, using the duplicate headers and #2759 as the initial clues. Reproduce the request against Elasticsearch 9.5.2 with the listed client versions; done means a bulk request sends only one compatible Accept header and no longer receives the 400 response.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, ruby
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.