elastic / elastic/docs-content

[Community]: Document ingesting Apache APISIX access logs into Elasticsearch

Open
#7,926 1 comment 0 reactions 0 assignees View on GitHub
community Team:SKI
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 17h
Merged PRs (30d)
130

Description

### What documentation page or section is affected

The general [Ingest: Bring your data to Elastic](https://www.elastic.co/docs/manage-data/ingest) section, particularly time-series ingestion.

The existing [Ingesting data from applications](https://www.elastic.co/docs/manage-data/ingest/ingesting-data-from-applications) section might also be relevant, although its current tutorials are scoped to Elastic Cloud Hosted and Elastic Cloud Enterprise. This request is for new content rather than a correction to an existing page.

### What should change?

Would the Elastic docs team accept a focused tutorial showing how to send structured API gateway access logs from Apache APISIX directly to Elasticsearch with the native APISIX `elasticsearch-logger` plugin?

Apache APISIX 3.17.0 serializes access-log entries as NDJSON and sends them to the Elasticsearch Bulk API. Its test suite includes Elasticsearch 9.0.2 as well as earlier major versions. The proposed tutorial would be written for Elastic readers and would cover:

- version-pinned prerequisites for Apache APISIX and a maintainer-approved Elastic Stack 9.x release;
- preparing a regular index and scoped credentials for the access logs;
- configuring APISIX to use HTTPS with certificate verification and authenticated Bulk requests;
- defining a minimal log format that includes `@timestamp` and a small set of useful request and response fields;
- sending requests through APISIX and verifying the indexed documents in Elasticsearch;
- explaining APISIX batching, retry, buffer, and sensitive-body logging boundaries;
- testing and documenting Bulk request and item-level failure behavior without claiming lossless or exactly-once delivery.

This would document direct Bulk API ingestion. It would not present APISIX as an Elastic Integration package, and it would not describe the current APISIX output as a data-stream integration because the plugin currently uses the Bulk `index` action.

Before opening a documentation PR, I would run and publish a version-pinned end-to-end validation. I would also confirm the AI-tool policy that applies to an external contributor and follow the Elastic tutorial structure, cumulative-docs rules, MyST syntax, Vale checks, CLA, and generative-AI disclosure requirements.

Would this topic fit the Elastic documentation? If so, which page or section should be its canonical home, and which Elastic Stack 9.x version should the example target?

### Additional info

Primary implementation evidence:

- [APISIX 3.17.0 `elasticsearch-logger` documentation](https://github.com/apache/apisix/blob/3.17.0/docs/en/latest/plugins/elasticsearch-logger.md)
- [APISIX 3.17.0 `elasticsearch-logger` implementation](https://github.com/apache/apisix/blob/3.17.0/apisix/plugins/elasticsearch-logger.lua)
- [APISIX 3.17.0 Elasticsearch test services](https://github.com/apache/apisix/blob/3.17.0/ci/pod/docker-compose.plugin.yml)
- [Elasticsearch Bulk API response and item-level error contract](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk-2)

One behavior would be called out explicitly in the guide: APISIX 3.17.0 currently treats a Bulk HTTP `200` response as request-level success and does not inspect the response's `errors` field or individual item statuses. If the docs team considers item-level handling a prerequisite for publishing this workflow, I would first take that requirement back to the APISIX community rather than document a stronger delivery guarantee than the implementation provides.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the affected Ingest pages and the linked Apache APISIX 3.17.0 plugin documentation, implementation, and test services, then review the Elasticsearch Bulk API contract. Confirm the canonical Elastic docs location, target Elastic Stack 9.x version, contributor policies, and an end-to-end validation covering the listed configuration and failure behavior. Done means an approved, version-pinned tutorial that does not claim unsupported delivery guarantees.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.