[Feature] Trace Pipeline Support in BanyanDB
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
## 🌟 Feature Request: Trace Pipeline Support in BanyanDB (Target: v0.11)
### **Summary**
Introduce a **trace pipeline framework** into BanyanDB to enable **trace-oriented processing**, **sampling**, and **enrichment** directly within the database layer.
The pipeline will serve as the foundation for intelligent, context-aware trace handling—allowing dynamic update, analysis, and retention decisions while preserving schema and performance guarantees.
---
### **Background**
BanyanDB currently supports efficient ingestion and querying of trace data, but processing is span-oriented and lacks capabilities for post-ingestion, trace-level operations.
Modern observability scenarios—such as **tail-based sampling**, **cross-service trace enrichment**, or **behavioral trace analysis**—require the database to treat a group of spans (sharing the same trace ID) as one logical unit.
Introducing a trace pipeline provides a structured mechanism to:
- Analyze and modify trace data after ingestion.
- Aggregate spans consistently, regardless of their reported order or source.
- Make smarter data retention decisions based on complete trace context.
---
### **Goal**
Deliver a **trace-level pipeline mechanism** that:
1. Processes collected spans as unified trace entities in best-effort strategy(No 100% guaranteed).
2. Supports runtime **enrichment and analysis** of existing tag values, further considering to declear trace-level schema, e.g. trace latency.
3. Maintains persistent **file-based processor context** across ingestion batches for continuity and scale.
4. Enables **tail-based sampling** via a reserved deletion/retention flag.
5. Integrates cleanly with **OAP** for schema declaration, dynamic loading, and test validation.
### Use case
This issue includes several key features and use cases, mostly around trace post-analysis, e.g., tail sampling.
Most things were discussed between @hanahmily and me at a high level. No detail is confirmed yet.
### Related issues
_No response_
### Are you willing to submit a pull request to implement this on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.