apache / apache/skywalking

[Feature] Trace Pipeline Support in BanyanDB

Open
#13,634 1 comment 2 reactions 3 assignees Claimed by @hanahmily View on GitHub
database feature
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.

### Description

## 🌟 Feature Request: Trace Pipeline Support in BanyanDB (Target: v0.11)

### **Summary**
Introduce a **trace pipeline framework** into BanyanDB to enable **trace-oriented processing**, **sampling**, and **enrichment** directly within the database layer.
The pipeline will serve as the foundation for intelligent, context-aware trace handling—allowing dynamic update, analysis, and retention decisions while preserving schema and performance guarantees.

---

### **Background**
BanyanDB currently supports efficient ingestion and querying of trace data, but processing is span-oriented and lacks capabilities for post-ingestion, trace-level operations.

Modern observability scenarios—such as **tail-based sampling**, **cross-service trace enrichment**, or **behavioral trace analysis**—require the database to treat a group of spans (sharing the same trace ID) as one logical unit.

Introducing a trace pipeline provides a structured mechanism to:
- Analyze and modify trace data after ingestion.
- Aggregate spans consistently, regardless of their reported order or source.
- Make smarter data retention decisions based on complete trace context.

---

### **Goal**
Deliver a **trace-level pipeline mechanism** that:
1. Processes collected spans as unified trace entities in best-effort strategy(No 100% guaranteed).
2. Supports runtime **enrichment and analysis** of existing tag values, further considering to declear trace-level schema, e.g. trace latency.
3. Maintains persistent **file-based processor context** across ingestion batches for continuity and scale.
4. Enables **tail-based sampling** via a reserved deletion/retention flag.
5. Integrates cleanly with **OAP** for schema declaration, dynamic loading, and test validation.

### Use case

This issue includes several key features and use cases, mostly around trace post-analysis, e.g., tail sampling.

Most things were discussed between @hanahmily and me at a high level. No detail is confirmed yet.

### Related issues

_No response_

### Are you willing to submit a pull request to implement this on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.