ozontech / ozontech/file.d

Bug:

Open
#898 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
504
Forks
263
Avg merge
4d 2h
Merged PRs (30d)
5

Description

Describe the bug
When using output.elasticsearch with batch_op_type: create (required for Elasticsearch/OpenSearch data streams), file.d reports errors even though documents are successfully created.
Error message:

unknown elasticsearch response, 'index' field in the response is empty

This happens because bulk responses contain "create": {...} instead of "index": {...}, which the plugin does not currently parse.

To Reproduce
Steps to reproduce the behavior:

Configure output.elasticsearch with batch_op_type: create.

Send logs to an Elasticsearch/OpenSearch data stream (index name starts with .ds-).

Observe file.d logs:

unknown elasticsearch response, 'index' field in the response is empty
some events from batch aren't written

Expected behavior
file.d should correctly parse bulk responses containing "create": {...} when batch_op_type: create is used, and treat them as successful writes.

Additional context

Version: file.d [v0.61.0]

Platform: Kubernetes

Elasticsearch/OpenSearch: OpenSearch [2.18]

Example response from server:

{"create":{
"_index":".ds-file-d-dev-cluster-apps-order-service-000069",
"_id":"ymfJcJkBL5nPyj0lnB8M",
"_version":1,
"result":"created",
"_shards":{"total":2,"successful":2,"failed":0},
"_seq_no":2826141,
"_primary_term":1,
"status":201
}}
Just to clarify: it’s not only about suppressing error logs.
Right now some events are actually marked as failed, so logs from search-gateway (and potentially other services) do not appear in OpenSearch when writing into data streams.

We need proper support for bulk responses with "create": {...} in output.elasticsearch (when batch_op_type: create is used).
Suppressing the error is not enough — the plugin should treat these responses as successful so that all events are indexed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the output.elasticsearch handling for batch_op_type: create and reproduce the issue with an OpenSearch data stream using the response example in the report. Verify that bulk responses containing a create object are parsed as successful writes, with no failed events or misleading error logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
backend, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.