can't specify pipeline for module
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
pipeline setting in the nginx module has no effect.
Elastic version: 7.4.0
Filebeat versions: 7.4.1
OS: Ubuntu 18.04
I have only one output defined in the "filebeat.yml":
```
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["elk.slavikf.com:9200"]
```
And then I'm trying to use specific pipeline by nginx module, so here is nginx module config:
```
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/*/*access.log"]
# Convert the timestamp to UTC
var.convert_timezone: true
# The Ingest Node pipeline ID associated with this input. If this is set, it
# overwrites the pipeline option from the Elasticsearch output.
output.elasticsearch.pipeline: 'filebeat-nginx-access-default'
pipeline: 'filebeat-nginx-access-default'
```
that doesn't work.
Here is what I see in the logs:
```
2019-10-31T07:35:32.072Z INFO elasticsearch/client.go:743 Attempting to connect to Elasticsearch version 7.4.0
2019-10-31T07:35:32.072Z DEBUG [elasticsearch] elasticsearch/client.go:761 GET http://elk.slavikf.com:9200/_xpack?human=false
2019-10-31T07:35:32.237Z DEBUG [modules] fileset/pipelines.go:67 Required processors: [{user_agent ingest-user-agent} {geoip ingest-geoip}]
2019-10-31T07:35:32.237Z DEBUG [elasticsearch] elasticsearch/client.go:761 GET http://elk.slavikf.com:9200/_nodes/ingest
2019-10-31T07:35:32.329Z DEBUG [elasticsearch] elasticsearch/client.go:761 GET http://elk.slavikf.com:9200/_ingest/pipeline/filebeat-7.4.1-nginx-access-default
2019-10-31T07:35:32.425Z DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.4.1-nginx-access-default already loaded
2019-10-31T07:35:32.426Z DEBUG [modules] fileset/pipelines.go:67 Required processors: []
2019-10-31T07:35:32.426Z DEBUG [elasticsearch] elasticsearch/client.go:761 GET http://elk.slavikf.com:9200/_ingest/pipeline/filebeat-7.4.1-nginx-error-pipeline
2019-10-31T07:35:32.520Z DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.4.1-nginx-error-pipeline already loaded
```
so, it is using `filebeat-7.4.1-nginx-access-default` pipeline instead of 'filebeat-nginx-access-default'
I also tried this options:
- output.elasticsearch.pipeline.id: 'filebeat-nginx-access-default'
- pipeline.id: 'filebeat-nginx-access-default'
same result
So, is there a way to tell nginx access module to use specific pipeline?
Contributor guide
Assessment
This issue has not been assessed yet.