Bug:
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 504
- Forks
- 263
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 5
Description
Describe the bug
When using output.elasticsearch with batch_op_type: create (required for Elasticsearch/OpenSearch data streams), file.d reports errors even though documents are successfully created.
Error message:
unknown elasticsearch response, 'index' field in the response is empty
This happens because bulk responses contain "create": {...} instead of "index": {...}, which the plugin does not currently parse.
To Reproduce
Steps to reproduce the behavior:
Configure output.elasticsearch with batch_op_type: create.
Send logs to an Elasticsearch/OpenSearch data stream (index name starts with .ds-).
Observe file.d logs:
unknown elasticsearch response, 'index' field in the response is empty
some events from batch aren't written
Expected behavior
file.d should correctly parse bulk responses containing "create": {...} when batch_op_type: create is used, and treat them as successful writes.
Additional context
Version: file.d [v0.61.0]
Platform: Kubernetes
Elasticsearch/OpenSearch: OpenSearch [2.18]
Example response from server:
{"create":{
"_index":".ds-file-d-dev-cluster-apps-order-service-000069",
"_id":"ymfJcJkBL5nPyj0lnB8M",
"_version":1,
"result":"created",
"_shards":{"total":2,"successful":2,"failed":0},
"_seq_no":2826141,
"_primary_term":1,
"status":201
}}
Just to clarify: it’s not only about suppressing error logs.
Right now some events are actually marked as failed, so logs from search-gateway (and potentially other services) do not appear in OpenSearch when writing into data streams.
We need proper support for bulk responses with "create": {...} in output.elasticsearch (when batch_op_type: create is used).
Suppressing the error is not enough — the plugin should treat these responses as successful so that all events are indexed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the output.elasticsearch handling for batch_op_type: create and reproduce the issue with an OpenSearch data stream using the response example in the report. Verify that bulk responses containing a create object are parsed as successful writes, with no failed events or misleading error logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100