elastic / elastic/docs-content

[REQUEST]: Add "Elastic Agent to Kafka to Kafka ES Sink to Elasticsearch" architecture

Open
#7,765 2 comments 0 reactions 0 assignees View on GitHub
Team:SKI
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

### Description

We have `Elastic Docs ›Elastic Ingest Reference Architectures [8.16] ›Elastic Agent to Elasticsearch with Kafka as middleware message queue
Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue` at . However, Logstash is not necessary between Elastic Agent as it can ship directly to Kafka. Please add this to this to the documentation.

Also please add the configuration for Kafka ES Sink that will pass through the collected data types (e.g logs, metrics, etc.) to the correct data stream and ingest pipelines. This is a critical piece of information for understanding how to set this all up.

Also, could add the link to the Sink documentation on page as well.
![Image](https://github.com/user-attachments/assets/90e43676-409a-4521-9dc7-6b3659ff6f63)

### Resources

- https://www.elastic.co/guide/en/ingest/current/agent-kafka-essink.html
- https://www.elastic.co/blog/index-data-elasticsearch-confluent-cloud
- https://www.elastic.co/blog/elastic-agent-output-kafka-data-collection-streaming
- https://www.elastic.co/blog/generative-ai-applications-confluent-cloud-apache-flink-elasticsearch
- https://docs.confluent.io/cloud/current/connectors/cc-elasticsearch-service-sink.html
- https://docs.confluent.io/cloud/current/connectors/limits.html#elasticsearch-service-sink-limits
- https://www.elastic.co/blog/elastic-agent-send-process-any-data-anywhere
- https://www.elastic.co/guide/en/fleet/current/kafka-output-settings.html

### Collaboration

Other (please describe below in the Point of contact section).

### Point of contact.

**Main contact:** @

**Stakeholders:**
- @a03nikki (I can help review the documentation as I have a user who is looking for this specifically)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.