Create a document-oriented file input for ingesting non-append file-based documents
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
**Describe the enhancement:**
Introduce a document‑oriented file input for Elastic Agent that is designed for ingesting non‑append, file‑based documents (e.g., XML, JSON, YAML, even TXT) rather than log streams.
Unlike filestream, this input would treat each file as a single logical document and re‑read the full file when changes are detected. Change detection could be based on configurable mechanisms such as file modification time, size, or checksum/hash. The input would emit one event per file and support update and delete semantics when files are modified or removed.
This enhancement would explicitly target document‑style ingestion and would not replace or modify filestream, which remains optimized for append‑only log workloads.
**Describe a specific use case for the enhancement or feature:**
Users want to index structured files from a filesystem (e.g., XML configuration files, reference datasets, metadata exports) so they are searchable in Elasticsearch. These files are often modified in place, not appended to.
Today, users sometimes attempt to use filestream for this purpose, which leads to missed updates or inconsistent ingestion because partial file modifications are not supported by design. A document‑oriented file input would provide a clear, supported ingestion path for “one file = one document” workflows and reduce misuse of log‑centric inputs for content ingestion.
**What is the definition of done?**
1. A new Elastic Agent input (or integration) exists that:
- Treats each file as a single event/document
- Re‑reads the full file when a change is detected
- Reliably detects in‑place file modifications
2. Change detection mechanism is configurable (e.g., mtime, size, checksum)
3. Supports delete semantics when files are removed
4. Has documented guardrails (e.g., file size limits, polling behavior)
5. Documentation clearly positions this input as document‑oriented and distinct from filestream
6. A basic example configuration and usage guide are provided
Contributor guide
Assessment
This issue has not been assessed yet.