elastic / elastic/ecs-dotnet

Adopt elastic-ingest-dotnet WiredStream logs.ecs / logs.otel endpoints

Open
#570 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
137
Forks
70
PR merge metrics
No merged PRs in 30d

Description

## Context

[`elastic-ingest-dotnet`](https://github.com/elastic/elastic-ingest-dotnet) now supports dedicated Kibana **wired Streams** bulk endpoints via `[WiredStream]` and `WiredStreamIngestEndpoint`:

| Endpoint | Bulk path | Intended payload |
|----------|-----------|------------------|
| `Logs` (default) | `logs/_bulk` | Legacy/generic wired path |
| `LogsEcs` | `logs.ecs/_bulk` | ECS field names stored as-is |
| `LogsOtel` | `logs.otel/_bulk` | OTel semantic conventions (+ ECS aliases) |

PR (ingest-dotnet): https://github.com/elastic/elastic-ingest-dotnet/pull/new/feature/streams
Docs (once merged): index-management → Streams / Wired streams / ECS and OTel endpoints
Product docs: [Get data into Streams](https://www.elastic.co/docs/solutions/observability/streams/get-data-in), [Wired streams field naming](https://www.elastic.co/docs/solutions/observability/streams/wired-streams-field-naming)

Today, ecs-dotnet shippers (`Elastic.Serilog.Sinks`, `Elastic.NLog.Targets`, `Elastic.Extensions.Logging`) write via classic **data streams** (`EcsDataStreamChannel` / index patterns) with optional template bootstrap. That maps to Kibana **classic** streams, not the managed **wired** path.

## Proposal

Move ecs-dotnet logging sinks onto the new ingest-dotnet wired APIs where appropriate:

1. **Add a wired Streams option** on sink/provider options (e.g. `WiredStream` / `IngestEndpoint = LogsEcs`) that uses `IngestChannel` + `[WiredStream]` (or the equivalent strategy factory) instead of only classic `DataStreamChannel`.
2. **Default recommendation for ECS JSON**: prefer `logs.ecs` so field naming matches [Elastic.CommonSchema](https://github.com/elastic/ecs-dotnet/tree/main/src/Elastic.CommonSchema) output without OTel conversion.
3. **Keep classic data-stream mode** for self-managed / existing deployments that still bootstrap ECS component templates.
4. **Document** classic vs wired in sink READMEs and link to ingest-dotnet Streams docs.
5. Optionally expose `LogsOtel` for callers that want OTel storage with ECS aliases.

## Non-goals (for this issue)

- Authoring Streamlang, significant events, or knowledge indicators from .NET (those remain Kibana Streams APIs/UI).
- Changing ECS document shape itself.

## Acceptance criteria

- [ ] Serilog / NLog / MEL shippers can target `logs.ecs` (and optionally `logs.otel`) wired ingest.
- [ ] Classic data-stream + bootstrap path remains available and documented.
- [ ] Docs explain when to choose wired vs classic for Elastic Observability Streams.
- [ ] Depend on a released `Elastic.Ingest.Elasticsearch` / `Elastic.Mapping` version that includes `WiredStreamIngestEndpoint`.

## References

- ingest-dotnet branch: `feature/streams`
- [Streams overview](https://www.elastic.co/docs/solutions/observability/streams/streams)

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the Elastic.Serilog.Sinks, Elastic.NLog.Targets, and Elastic.Extensions.Logging entry points, comparing their EcsDataStreamChannel/DataStreamChannel paths with IngestChannel, WiredStream, and WiredStreamIngestEndpoint. Verify the released Elastic.Ingest.Elasticsearch and Elastic.Mapping versions first. Done means wired logs.ecs support exists for all three shippers, optional logs.otel is addressed, classic mode remains available, and the READMEs explain both choices.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, elasticsearch
Domain
backend, documentation, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.