opensearch-project / opensearch-project/data-prepper
Allow trace group for partial traces
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
DataPrepper does an aggregation of all spans with the same trace id. If encounters a span with a null parent span id, it assigns its name as trace group to all spans. This allows classification in the OpenSearch Dashboards observability plugin. However, this approach fails, if the global parent span does not arrive in time or at all. Consider the following situation:
In this picture, DataPrepper receives all coloured spans. It does not receive the gray spans. This might be because they are created in another system outside of the reach of the observability infrastructure the DataPrepper instance belongs to. This can be another vendor or a client system where the coloured spans are generated within a SaaS solution.
Currently, DataPrepper will not create a trace group entry for the spans, since the global trace parent is never received.
Describe the solution you'd like
It would be great, if in that case DataPrepper would follow the connection along the parent span ids until it can no longer resolve the parent. If this leads to a unique span, this span should be used as the trace parent instead of the original global trace parent.
The picture shows a conflict situation, where no unique parent can be determined. In that case, no trace group should be issued, keeping the current behaviour.
Additional context
For the implementation, this feature could be an option in the OTelTraceRawProcessor where the detection of a parent span needs to be changed.
Alternatively, in the OTelTraceGroupProcessor the search query could be changed.
It would also be possible to create a new processor or action in the aggregate processor, that fills in empty trace groups if possible.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with parent detection in data-prepper-plugins/otel-trace-raw-processor/.../OTelTraceRawProcessor.java and the search query in data-prepper-plugins/otel-trace-group-processor/.../OTelTraceGroupProcessor.java. Decide which processor or configuration should own unresolved-parent traversal, then verify that a unique unresolved ancestor produces a trace group while conflicting ancestors do not.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100