Adopt elastic-ingest-dotnet WiredStream logs.ecs / logs.otel endpoints
- Dominant language
- HTML
- Stars
- 137
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
## Context
[`elastic-ingest-dotnet`](https://github.com/elastic/elastic-ingest-dotnet) now supports dedicated Kibana **wired Streams** bulk endpoints via `[WiredStream]` and `WiredStreamIngestEndpoint`:
| Endpoint | Bulk path | Intended payload |
|----------|-----------|------------------|
| `Logs` (default) | `logs/_bulk` | Legacy/generic wired path |
| `LogsEcs` | `logs.ecs/_bulk` | ECS field names stored as-is |
| `LogsOtel` | `logs.otel/_bulk` | OTel semantic conventions (+ ECS aliases) |
PR (ingest-dotnet): https://github.com/elastic/elastic-ingest-dotnet/pull/new/feature/streams
Docs (once merged): index-management → Streams / Wired streams / ECS and OTel endpoints
Product docs: [Get data into Streams](https://www.elastic.co/docs/solutions/observability/streams/get-data-in), [Wired streams field naming](https://www.elastic.co/docs/solutions/observability/streams/wired-streams-field-naming)
Today, ecs-dotnet shippers (`Elastic.Serilog.Sinks`, `Elastic.NLog.Targets`, `Elastic.Extensions.Logging`) write via classic **data streams** (`EcsDataStreamChannel` / index patterns) with optional template bootstrap. That maps to Kibana **classic** streams, not the managed **wired** path.
## Proposal
Move ecs-dotnet logging sinks onto the new ingest-dotnet wired APIs where appropriate:
1. **Add a wired Streams option** on sink/provider options (e.g. `WiredStream` / `IngestEndpoint = LogsEcs`) that uses `IngestChannel` + `[WiredStream]` (or the equivalent strategy factory) instead of only classic `DataStreamChannel`.
2. **Default recommendation for ECS JSON**: prefer `logs.ecs` so field naming matches [Elastic.CommonSchema](https://github.com/elastic/ecs-dotnet/tree/main/src/Elastic.CommonSchema) output without OTel conversion.
3. **Keep classic data-stream mode** for self-managed / existing deployments that still bootstrap ECS component templates.
4. **Document** classic vs wired in sink READMEs and link to ingest-dotnet Streams docs.
5. Optionally expose `LogsOtel` for callers that want OTel storage with ECS aliases.
## Non-goals (for this issue)
- Authoring Streamlang, significant events, or knowledge indicators from .NET (those remain Kibana Streams APIs/UI).
- Changing ECS document shape itself.
## Acceptance criteria
- [ ] Serilog / NLog / MEL shippers can target `logs.ecs` (and optionally `logs.otel`) wired ingest.
- [ ] Classic data-stream + bootstrap path remains available and documented.
- [ ] Docs explain when to choose wired vs classic for Elastic Observability Streams.
- [ ] Depend on a released `Elastic.Ingest.Elasticsearch` / `Elastic.Mapping` version that includes `WiredStreamIngestEndpoint`.
## References
- ingest-dotnet branch: `feature/streams`
- [Streams overview](https://www.elastic.co/docs/solutions/observability/streams/streams)
Contributor guide
Research direction
Start by inspecting the Elastic.Serilog.Sinks, Elastic.NLog.Targets, and Elastic.Extensions.Logging entry points, comparing their EcsDataStreamChannel/DataStreamChannel paths with IngestChannel, WiredStream, and WiredStreamIngestEndpoint. Verify the released Elastic.Ingest.Elasticsearch and Elastic.Mapping versions first. Done means wired logs.ecs support exists for all three shippers, optional logs.otel is addressed, classic mode remains available, and the READMEs explain both choices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, elasticsearch
- Domain
- backend, documentation, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100