elastic / elastic/integrations

Migrate uses of the logfile input to filestream

Open
#2,518 5 comments 0 reactions 1 assignee Claimed by @flexitrev View on GitHub
draft enhancement Integration:filestream Integration:log Team:Elastic-Agent-Data-Plane technical-debt
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## Goal

The goal of this issue is to migrate existing packages that rely on `log` (`logfile`) input to `filestream`. Updating the package must be backwards compatible. The change in the integration package should be hidden from users.

The only user-visible change should be the value of `input.type` in the event from `log` to `filestream`.

### Why migrate?

The new `filestream` input has replaced the good, old `log` a.k.a. `logfile` input in Beats. The filestream input is GA since 7.16 and at the same time `logfile` was deprecated. In the last few releases, we added numerous bug fixes to the new input, and now we are working on enhancements. It is stable enough for adoption in Integrations.

It comes with several improvements over the old input: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream

### Differences

There are several differences in the configuration of the inputs:

- [`close.*` options](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-close-options)
- [`parsers` list](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_parsers) instead of top-level `json.*` and `multiline.*` settings
- [`scanner` options](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_scanner_options): excluding files, scan interval setting

### How to migrate integrations?

Some of the changes might be automated, for example, renaming `close_removed` to `close.on_state_change.removed`. But some options require manual checking and adjustments e.g. the parsing of lines. Also, there are new options, like `include_files` the counterpart of `exclude_files`. Those should be validated to see if existing configurations could be improved.

#### General steps for migrating a package
- [ ] Does it include options that can be replaced?
- `exclude_files` -> `prospector.scanner.exclude_files`
- `recursive_glob` -> `prospector.scanner.recursive_glob`
- `symlinks` -> `prospector.scanner.symlinks`
- `scan_interval` -> `prospector.scanner.interval`
- `close_inactive` -> `close.on_state_change.inactive`
- `close_renamed` -> `close.on_state_change.renamed`
- `close_removed` -> `close.on_state_change.removed`
- `close_eof` -> `close.reader.eof`
- `close_timeout` -> `close.reader.after_interval`
- `harvester_buffer_size` -> `buffer_size`
- `max_bytes` -> `message_max_bytes`
- [ ] Rewrite parsing configuration
- all parsers must go under `parsers` option
- order of parsing can be changed
- `json` is renamed to `ndjson`
- new parser called `container` is preferred over `container` input
- [ ] Can any of the new options be adopted?
- [`prospector.scanner.include_files`](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_prospector_scanner_include_files)

### How to migrate on Filebeat side?

If someone has been using e.g. Apache integrations and updates to the new version, input change must not be visible to users. Upgrading the package must not mean that the monitored files are read from the beginning. State information from the `log` input has to be passed to the `filestream` input, so it can continue where log input has left off. Given that position tracking is similar in the inputs, changing the state ID from `log::{id}::{device}-{inode}` to `filestream::{id}::{device}-{inode}` should work.

## Tasks

### Beats side migration

- [ ] support for reading state information of `log` input from `filestream` input

### Packages

This is the list of packages that use `logfile` input to collect data.

- [ ] activemq
- [ ] apache
- [ ] atlassian_bitbucket
- [ ] atlassian_confluence
- [ ] atlassian_jira
- [ ] auditd
- [ ] barracuda
- [ ] bbot
- [ ] bluecoat
- [ ] carbonblack_edr
- [ ] cassandra
- [ ] cef
- [ ] checkpoint
- [ ] cisco_aironet
- [ ] cisco_asa
- [ ] cisco_ftd
- [ ] cisco_ios
- [ ] cisco_meraki
- [ ] cisco_secure_email_gateway
- [ ] citrix_adc
- [ ] citrix_waf
- [ ] crowdstrike
- [ ] cyberarkpas
- [ ] cylance
- [ ] elasticsearch
- [ ] falco
- [ ] fireeye
- [ ] forcepoint_web
- [ ] fortinet_forticlient
- [ ] fortinet_fortiedr
- [ ] fortinet_fortigate
- [ ] haproxy
- [ ] hashicorp_vault
- [ ] ibmmq
- [ ] iis
- [ ] infoblox_nios
- [ ] iptables
- [ ] kafka
- [ ] kibana
- [ ] logstash
- [ ] mattermost
- [ ] microsoft_defender_endpoint
- [ ] microsoft_dhcp
- [ ] microsoft_exchange_online_message_trace
- [ ] microsoft_sqlserver
- [ ] modsecurity
- [ ] mongodb
- [ ] mysql
- [ ] nats
- [ ] netscout
- [ ] nginx
- [ ] oracle_weblogic
- [ ] osquery
- [ ] panw
- [ ] platform_observability
- [ ] postgresql
- [ ] pps
- [ ] rabbitmq
- [ ] radware
- [ ] redis
- [ ] santa
- [ ] snort
- [ ] sonicwall_firewall
- [ ] sophos
- [ ] stan
- [ ] suricata
- [ ] symantec_endpoint
- [ ] system
- [ ] thycotic_ss
- [ ] ti_recordedfuture
- [ ] tomcat
- [ ] traefik
- [ ] zeek

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.