NETWAYS / NETWAYS/ansible-collection-elasticstack

[Bug]: Beats setup tasks (filebeat/metricbeat) inconsistent with auditbeat

Open
#508 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug component:beats
Dominant language
Jinja
Stars
14
Forks
11
Avg merge
1d 47m
Merged PRs (30d)
6

Description

The three Beats run their setup task (loads index template, ILM, ingest pipelines, dashboards into Elasticsearch) inconsistently, and each has gaps. Auditbeat is the closest to correct and should be the reference for the other two.

Problems

  • metricbeat uses bare metricbeat setup (= index-management + dashboards + ML, but no pipelines), with a marker misleadingly named pipelines_created.
  • filebeat is not gated on the Elasticsearch output (fails on the default Logstash output); passes --pipelines --dashboards but no --index-management; modules live in modules.d/, so setup --pipelines finds none (needs --modules); marker is per-module.
  • auditbeat uses a run-once-forever marker, so modules/filesets added later never get their pipelines loaded.

Target

  • Gate all three on beats_<beat>_output == "elasticsearch" and run_once.
  • Always use explicit flags, never bare setup, so the task is self-documenting: setup --index-management --pipelines, plus --dashboards only when Kibana is configured. --index-management is safe (setup.template.overwrite: false).
  • Rename metricbeat's pipelines_created marker to an accurate name (e.g. setup_done).
  • Replace the run-once-forever marker with re-run on change: register the module/config task and run setup when: <register> is changed.
  • filebeat: add --modules (or adopt the inline data-driven design) so module pipelines actually load.
  • Drop "(experimental)" from the beats_filebeat_modules description.

Background

With the Elasticsearch output the index template and pipelines auto-load on first run anyway; with the Logstash output nothing auto-loads, so setup is only really needed there — yet the tasks gate on the ES output and skip exactly that case.

The module handling is redesigned data-driven in #510 / #505 / #516 (inline, one list = desired state), which simplifies this further (inline → setup --pipelines sees the modules directly, no --modules). This issue is the fix for the current architecture.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Beats setup tasks and the module/config tasks referenced in the issue, then review the data-driven redesign in #510, #505, and #516. Compare filebeat, metricbeat, and auditbeat against the explicit setup flags, output and run_once conditions. Done means all three setup flows use the requested gating, markers, rerun behavior, module handling, and description change.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, elasticsearch
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.