influxdata / influxdata/telegraf

A new plugin inputs.opensearch with the RemoteStats API

Open
#16,562 7 comments 1 reaction 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Hello Team,

Currently, the following inputs plugins are available for use with the OpenSearch service: `inputs.elasticsearch`, `inputs.elasticsearch_query`, `inputs.opensearch_query`. It is possible to use the elasticsearch plugins with the OpenSearch service with some caveats, as the APIs are aligned (new just need to replace appropriate strings).

While the OpenSearch service is being developed, a new API comes into play that we are also interested in scraping for the running nodes:

- [Segment Replication API](https://opensearch.org/docs/latest/api-reference/cat/cat-segment-replication/)
```
GET /_cat/segment_replication
GET /_cat/segment_replication/{index}
```

- [Remote Store Stats API](https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/)
```
GET _remotestore/stats/
GET _remotestore/stats//
```

I briefly checked Elasticsearch for the corresponding metrics API and it seems that there is no 1-to-1 match for the metrics we need. So I propose to add a new `opensearch` plugin to inputs, so we can query new metrics.

I have a draft of changes to demonstrate the possible solution of integrating new metrics to the elasticsearch plugin, please let me know if you are iterested in introducing a new inputs plugin and I need to submit a PR>

Configuration Example:
```
[[inputs.elasticsearch]]
tagexclude = []
servers = [ "http://127.0.0.1:9200",]
local = true
ccr_stats = true
cluster_health = true
cluster_health_level = "indices"
cluster_stats = true

[processors]
[[processors.strings]]
namepass = [ "elasticsearch_*",]
[[processors.strings.replace]]
measurement = "*"
old = "elasticsearch_"
new = "opensearch_"
```

### Expected behavior

A new `opensearch` plugin is added to handle OpenSearch API specifics.

### Actual behavior

Not able to scrape new opensearch metrics, need to replace `elasticsearch_` with `opensearch_`.

https://opensearch.org/docs/latest/api-reference/cat/cat-segment-replication/
https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Review the existing inputs.elasticsearch, inputs.elasticsearch_query, and inputs.opensearch_query plugins, then read the linked Segment Replication and Remote Store Stats API documentation. Define the new plugin's scope around collecting the requested OpenSearch metrics and verify that its configuration and measurements work against the documented endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.