elastic / elastic/docs-content

[REQUEST]: Improve Elastic Agent Logstash output guidance

Open
#7,764 3 comments 0 reactions 0 assignees View on GitHub
Team:SKI
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

### Description

When users are following the flow Elastic Agent > Logstash > Elasticsearch, we should be more "explicit" on some sections.

## Problems

1. Without a running Kibana Fleet instance and creating a Logstash output in the Fleet UI, one doesn't know what are the permissions to grant to Logstash in order to write data
2. It's not obvious the API Key or user to use in the ES Output of Logstash must be created with specific permissions (1) and must be created on the destination cluster which might not be the same cluster where Fleet is "setup"
3. It's not obvious that the integration assets must be installed in the destination ES cluster, which might not be the same cluster where Fleet is "setup"

## Possible improvements

The page [here](https://www.elastic.co/guide/en/fleet/current/ls-output-settings.html#:~:text=The%20Logstash%20pipeline%20definition%20below%20is%20an%20example.%20Please%20refer%20to%20the%20Additional%20Logstash%0Aconfiguration%20required%20steps%20when%20creating%20the%20Logstash%20output%20in%20the%20Fleet%20outputs%20page.) should highlight that the user or API key to use in the Elasticsearch output of Logstash must:
- Be created on the destination Elasticsearch cluster (which might not be the same cluster where Fleet is installed/setup)
- It would be ideal to provide the permissions to grant without the need to having a running functional Kibana Fleet UI. The API Key creation will grant by default (on 8.17). Ideally this should be generated from the code or from the manifests, we should not hardcode it as it will go stale.
```
"logstash-output": {
"cluster": [
"monitor"
],
"indices": [
{
"names": [
"logs-*-*",
"metrics-*-*",
"traces-*-*",
"synthetics-*-*",
".logs-endpoint.diagnostic.collection-*",
".logs-endpoint.action.responses-*",
"profiling-*",
".profiling-*"
],
"privileges": [
"auto_configure",
"create_doc"
],
"allow_restricted_indices": false
}
],
"applications": [],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
}
}
}
```
- Add a reminder about the fact the integration assets used in the policy shipping data through Logstash and then Elasticsaerch must be installed on the destination ES cluster (which sometimes is not the same where Fleet is installed)

### Resources

-

### Collaboration

TBD. The docs and product team will work together to determine the best path forward.

### Point of contact.

**Main contact:** @lucabelluccini

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.