elastic / elastic/beats

Separate "Metricbeat index aliases" from "xpack.enabled fork removal", revert fork removal

Open
#26,480 8 comments 0 reactions 1 assignee Claimed by @sayden View on GitHub
Feature:Stack Monitoring Team:Integrations Team:Obs-DC
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

UPDATED: 28-10-2021

Related: https://github.com/elastic/beats/pull/24427

Original Context

In this PR: https://github.com/elastic/beats/pull/19747, we outlined a number of requirements for Stack Monitoring data collection.

> This PR prepares the `elasticsearch/node_stats` metricset (without setting `xpack.enabled: true`) to collect and index data for the Stack Monitoring UI. Concretely, it does the following:
> - introduces new fields, collecting data needed by the Stack Monitoring UI. These fields are named following [Metricbeat](https://www.elastic.co/guide/en/beats/devguide/current/event-conventions.html)/[ECS](https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html#_guidelines_for_field_names) naming conventions.
> - introduces field aliases from current fields known to the Stack Monitoring UI to the fields defined by the metricset. This will aid transitioning the Stack Monitoring UI in 7.x to be driven off the metricset with `xpack.enabled` omitted. In 8.0 the aliases will be removed and the Stack Monitoring UI code will have to be overhauled to reference the new fields directly.
> - removes the `data_xpack.go` file implementing the `xpack.enabled: true` code path for the `node_stats` metricset. Going forward, the `node_stats` metricset will simply ignore the value of the `xpack.enabled` setting.

Points (1) and (2) mentioned here involve making sure that the various Stack Monitoring Metricbeat modules store data "correctly" when `xpack.enabled: false` (the default value), i.e. using ECS paths for all necessary fields and stored in the `metricbeat-*` indices. It also involves creating the necessary index aliases from the "legacy" paths to the "correct" ECS paths where the data will now live, so that a query to `metricbeat-*` looking for `my.old.field.path` will be silently redirected behind the scenes to return the value from `metricbeat-*` : `ecs.path.to.same.field`.

That work is required to ship in the released modules ASAP so that we can test out the aliases that are in place with real data as easily as possible.

Point (3) above is about removing the `xpack.enabled` fork altogether, i.e. "the `node_stats` metricset will simply ignore the value of the `xpack.enabled` setting". We don't want to ship this because this will mean that users who have opted into the `xpack.enabled` experience (i.e. most if not all of our Stack Monitoring users, because this is the currently recommended way to use Metricbeat collection for Stack Monitoring) will suddenly have their data written to a new index and any visualizations will need to be updated to point to the `metricbeat-*` index (the paths are redirected but the index must be updated). In addition, `metricbeat-*` indices do not apply any ILM policies to these data in the same way that the `.monitoring` indices do, so this change would silently begin storing data for a different amount of time which could incur costs for the user without them knowing, etc.

For this reason, we need to separate these two sets of changes so that we can backport/merge points (1) and (2) above, while reverting (3) _only_.

**NOTE: We have made the decision to continue supporting internal collection and metricbeat (xpack.enabled: true) collection in 8.0.0 and beyond, which makes this ticket even more important.**

Previous AC

AC:
* When `xpack.enabled` is set to false (or not set at all), the Elasticsearch, Kibana, Logstash, and Beats modules each store their data in ECS-appropriate fields, with aliases from the legacy fields to the new ECS fields installed as well.
* When `xpack.enabled` is set to true for any of these modules, they will continue to do what they have done in the past, i.e. store their data in the `.monitoring-*` indices using the legacy paths.
* When Elastic Agent runs metricbeat, it should set `xpack.enabled` to false and not provide that option to the user at all, and should write to an appropriately named `metrics-*` datastream that has the mentioned field aliases installed in its mappings.

AC:
* When a user is running Metricbeat Standalone in 8.0+
- With `xpack.enabled` set to true for a given module, that module's data is queryable from the `.monitoring-{module}-mb*` indices and can be queried using the "non-ECS" paths as filters, aggregations, etc.
- With `xpack.enabled` not set, or set to false for a given module, that module's data does whatever you all think is best (we don't mind if this data is not visible in the Stack Monitoring UI, because they haven't been before)
* When a user is running Agent (with Metricbeat inside) in 8.0+
- There is no way for that user to set `xpack.enabled` at all for these integrations/packages
- The data is written to the relevant data stream for that integration/package
- These data can be queried using the "non-ECS" paths as filters, aggregations, etc.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.