opensearch-project / opensearch-project/data-prepper
Conditional Routing for service-map data is not working.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Issue:
We're implementing a conditional based routing for trace data for creating dynamic indexes(multiple sinks) based upon one of the field present in the dataset (tenant_name). The conditional routing is working for traces, but not for the service-map data. The template for the service-map doesn't contain more fields and adding of custom fields is also not possible. So the service-map data is not going to relevant dynamic indexes like the trace data, which is causing issues while stitching the data at user interface level.
Reproduce:
Have multiple sinks for trace & service-map data depending upon the tenant_name (tenant1, tenant2 & tenant3 ... tenantN) field.
The service-map data won't follow the same routing like trace data.
Expected behavior
The trace & service-map data need to follow the similar kind of dynamic routing.
Config: pipelines.yaml for trace & service-map data
################################################################################
traces-entry-pipeline:
workers: 1
delay: "100"
source:
otel_trace_source:
ssl: false
buffer:
bounded_blocking:
route:
- tenant1_traces_svcmaps: '/attributes/span.attributes.tenant_name == "tenant1"'
- tenant2_traces_svcmaps: '/attributes/span.attributes.tenant_name == "tenant2"'
- tenant3_traces_svcmaps: '/attributes/span.attributes.tenant_name == "tenant3"'
sink:
- stdout:
- pipeline:
name: "traces-svcmaps-pipeline-t1"
routes: [tenant1_traces_svcmaps]
- pipeline:
name: "traces-svcmaps-pipeline-t2"
routes: [tenant2_traces_svcmaps]
- pipeline:
name: "traces-svcmaps-pipeline-t3"
routes: [tenant3_traces_svcmaps]
traces-svcmaps-pipeline-t1:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-entry-pipeline"
buffer:
bounded_blocking:
sink:
- stdout:
- pipeline:
name: "trace-raws-pipeline-t1"
- pipeline:
name: "service-maps-pipeline-t1"
traces-svcmaps-pipeline-t2:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-entry-pipeline"
buffer:
bounded_blocking:
sink:
- stdout:
- pipeline:
name: "trace-raws-pipeline-t2"
- pipeline:
name: "service-maps-pipeline-t2"
traces-svcmaps-pipeline-t3:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-entry-pipeline"
buffer:
bounded_blocking:
sink:
- stdout:
- pipeline:
name: "trace-raws-pipeline-t3"
- pipeline:
name: "service-maps-pipeline-t3"
trace-raws-pipeline-t1:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-svcmaps-pipeline-t1"
buffer:
bounded_blocking:
processor:
- otel_traces:
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
# cert: "/usr/share/data-prepper/opensearch.crt"
ssl_verification_enabled: false
insecure: true
username:
password:
index: acn_tenant1_traces-%{yyyy.MM.dd} # Dynamically creates index based on route
service-maps-pipeline-t1:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-svcmaps-pipeline-t1"
buffer:
bounded_blocking:
processor:
- service_map:
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
# cert: "/usr/share/data-prepper/opensearch.crt"
ssl_verification_enabled: false
insecure: true
username:
password:
index: acn_tenant1_servicemap-%{yyyy.MM.dd} # Dynamically creates index based on route
trace-raws-pipeline-t2:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-svcmaps-pipeline-t2"
buffer:
bounded_blocking:
# buffer_size: 128 # 10240
# batch_size: 8 # 160
processor:
- otel_traces:
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
# cert: "/usr/share/data-prepper/opensearch.crt"
ssl_verification_enabled: false
insecure: true
username:
password:
index: acn_tenant2_traces-%{yyyy.MM.dd} # Dynamically creates index based on route
service-maps-pipeline-t2:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-svcmaps-pipeline-t2"
buffer:
bounded_blocking:
processor:
- service_map:
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
# cert: "/usr/share/data-prepper/opensearch.crt"
ssl_verification_enabled: false
insecure: true
username:
password:
index: acn_tenant2_servicemap-%{yyyy.MM.dd} # Dynamically creates index based on route
trace-raws-pipeline-t3:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-svcmaps-pipeline-t3"
buffer:
bounded_blocking:
processor:
- otel_traces:
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
# cert: "/usr/share/data-prepper/opensearch.crt"
ssl_verification_enabled: false
insecure: true
username:
password:
index: acn_tenant3_traces-%{yyyy.MM.dd} # Dynamically creates index based on route
service-maps-pipeline-t3:
workers: 1
delay: "100"
source:
pipeline:
name: "traces-svcmaps-pipeline-t3"
buffer:
bounded_blocking:
processor:
- service_map:
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
# cert: "/usr/share/data-prepper/opensearch.crt"
ssl_verification_enabled: false
insecure: true
username:
password:
index: acn_tenant3_servicemap-%{yyyy.MM.dd} # Dynamically creates index based on route
################################################################################
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pipelines.yaml configuration and trace the routing into the service_map processor. Reproduce the tenant-based trace and service-map setup described in the issue, then verify that service-map data reaches the corresponding tenant-specific indexes just like trace data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100