elastic / elastic/beats

GA support for reading from journald

Open
#37,086 47 comments 1 reaction 1 assignee Claimed by @belimawr View on GitHub
Team:Elastic-Agent Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

- Relates https://github.com/elastic/elastic-agent/issues/3650

As of Debian 12 system logs are exclusively available via journald by default. Today we support reading journald logs via the Filebeat [journald input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-journald.html), which is still in technical preview and has several major bugs filed against it. See https://github.com/elastic/beats/issues?q=is%3Aissue+is%3Aopen+journald notably:

- [x] https://github.com/elastic/beats/issues/34077
- [x] https://github.com/elastic/beats/issues/32782
- [x] https://github.com/elastic/beats/issues/30398

We need to provide a GA way to read journald logs. There are two paths to this:

1. Fix the major issues in the journald input and GA it as is. All integrations that previously read syslog files by default will need a conditional to specify that journald should be used instead of one of the log files on Linux (see [example](https://github.com/elastic/integrations/blob/9dcd6eac196eed413eff898c1ecbc46fe1d745e1/packages/iptables/data_stream/log/agent/stream/journald.yml.hbs#L1). Possibly this conditional will need to be on the Linux distribution and not just Linux as a platform.
2. ~Fold the existing journald functionality into filestream, so that there is only one way to read log files and all existing uses of filestream to read system logs continue to work with no or minimal modification. In the ideal case we detect we are reading journald logs based on a .journal extension or well known file paths, but we may need a configuration flag for this. If we do end up with a configuration flag we could consider implementing journald support as a type of parser https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_parsers~

Edit:
Option 1 is the path forward, we'll keep the separate journald input.

To close this issue we'll need to:

```[tasklist]
### Must have
- [ ] https://github.com/elastic/integrations/issues/9690
- [x] https://github.com/elastic/beats/issues/34077
- [x] https://github.com/elastic/beats/issues/30398
- [ ] https://github.com/elastic/beats/issues/39352
- [ ] https://github.com/elastic/beats/issues/41083
- [ ] https://github.com/elastic/beats/issues/41353
- [ ] https://github.com/elastic/beats/issues/41378
- [ ] https://github.com/elastic/integrations/issues/10797
- [ ] https://github.com/elastic/beats/issues/42208
- [ ] https://github.com/elastic/beats/issues/41278
- [ ] https://github.com/elastic/integrations/issues/13019
- [ ] https://github.com/elastic/beats/issues/44040
```

```[tasklist]
### Nice to have
- [x] https://github.com/elastic/beats/issues/32782
- [ ] https://github.com/elastic/beats/issues/37877
- [ ] https://github.com/elastic/beats/issues/41331
- [ ] https://github.com/elastic/beats/issues/37876
- [ ] https://github.com/elastic/beats/issues/39791
- [ ] https://github.com/elastic/beats/issues/40185
- [ ] https://github.com/elastic/beats/issues/40478
- [ ] https://github.com/elastic/beats/issues/40479
- [ ] https://github.com/elastic/beats/issues/40526
- [ ] https://github.com/elastic/integrations/issues/10998
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.