fluent / fluent/fluent-plugin-opensearch
Opensearch version and distribution check fails if request path is set
- Dominant language
- Ruby
- Stars
- 64
- Forks
- 23
- Avg merge
- 1h 53m
- Merged PRs (30d)
- 3
Description
#### Steps to replicate
Set the `path` parameter in the output plugin config (the block) to post requests to a specific index URL.
It is necessary to be able to do this for access control, but it causes the version/distribution check to make it's request to the index instead of the base URL, and does not get the version and distribution parameters it is requesting:
> 2022-07-17 10:24:50 +0000 [error]: #0 unexpected error error_class=OpenSearch::UnsupportedProductError error="The client is not supported for the provided version and distribution combination."
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:78:in 'verify_version_and_distribution'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:74:in 'verify_open_search'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:47:in 'method_missing'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-api-1.0.0/lib/opensearch/api/actions/info.rb:45:in 'info'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:518:in 'detect_os_major_version'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:510:in 'block in handle_last_seen_os_major_version'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/opensearch_index_template.rb:70:in 'retry_operate'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:507:in 'handle_last_seen_os_major_version'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:383:in 'configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in 'configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:132:in 'add_match'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in 'block in configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in 'each'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in 'configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in 'configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in 'configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in 'run_configure'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:719:in 'block in run_worker'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in 'main_process'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in 'run_worker'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in ''
> 2022-07-17 10:24:50 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in 'require'
> 2022-07-17 10:24:50 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in 'require'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in ''
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in 'load'
> 2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in ''
#### Expected Behavior or What you need to ask
The plugin queries the base URL for version/distribution check, then changes to the specified path when posting write operations.
#### Using Fluentd and OpenSearch plugin versions
* OS version
5.4.181-99.354.amzn2.x86_64
* Bare Metal or within Docker or Kubernetes or others?
fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-opensearch-1.0
* Fluentd v1.0 or later
* paste result of ``fluentd --version`` or ``td-agent --version``
fluentd 1.14.6
* OpenSearch plugin version
* paste boot log of fluentd or td-agent
* paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
*** LOCAL GEMS ***
addressable (2.8.0)
aws-eventstream (1.2.0)
aws-partitions (1.574.0)
aws-sdk-core (3.130.0)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (2.2.24, default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
domain_name (0.5.20190701)
etc (default: 1.1.0)
excon (0.92.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
ffi (1.15.5)
ffi-compiler (1.0.1)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.14)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-json-in-json-2 (1.0.2)
fluent-plugin-kubernetes_metadata_filter (2.9.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.3)
fluent-plugin-parser-cri (0.1.1)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-systemd (1.0.5)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http (4.4.1)
http-accept (1.7.0)
http-cookie (1.0.4)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.1)
json (default: 2.3.0)
jsonpath (1.1.0)
kubeclient (4.9.3)
logger (default: 1.4.2)
lru_redux (1.1.0)
matrix (default: 0.2.0)
mime-types (3.4.1)
mime-types-data (3.2022.0105)
msgpack (1.5.1)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
netrc (0.11.0)
observer (default: 0.1.0)
oj (3.11.0)
open3 (default: 0.1.0)
opensearch-api (1.0.0)
opensearch-ruby (1.0.0)
opensearch-transport (1.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
prime (default: 0.1.1)
prometheus-client (4.0.0)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.6)
racc (default: 1.4.16)
rake (13.0.6)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
recursive-open-struct (1.1.3)
reline (default: 0.1.5)
rest-client (2.1.0)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.2.5)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
systemd-journal (1.4.2)
timeout (default: 0.1.0)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
unf (0.1.4)
unf_ext (0.0.8.1)
uri (default: 0.10.0)
webrick (1.7.0, default: 1.6.1)
yajl-ruby (1.4.2)
yaml (default: 0.1.0)
zlib (default: 1.1.0)
* OpenSearch version (optional)
1.3.3
Contributor guide
Assessment
This issue has not been assessed yet.