redpanda-data / redpanda-data/benthos

Streams mode: spans carry no stream identity

Open
#503 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
571
Forks
120
Avg merge
2d 1h
Merged PRs (30d)
18

Description

In streams mode every span is named after the component type (input_generate, mapping, output_sql_raw), so spans from different streams look identical in a tracing backend. There is no config-level way to tell them apart: label is not applied to spans (internal/component/input/async_reader.go, processor/auto_observed.go, output/async_writer.go build the name from the type only), and the tracer's tags are resource attributes shared by the whole process.

Logs and metrics already have this: Manager.forStream adds a stream field to the logger and a stream label to the metrics, but passes the tracer provider through untouched. The tracer is the odd one out.

Use case: one Redpanda Connect process running ~30 streams in streams mode, exporting to Tempo via OTLP. I want to write { span.stream = "knx" } in TraceQL, or filter on the stream in Jaeger. Today the only way is a group_by_value processor in each stream, which happens to set a group attribute on its span; that works, but it is a batching processor used for a side effect, and the attribute lands on a child span rather than the root.

Proposal: forStream wraps the tracer provider so that every span it starts carries stream=<id>, added as a span start option so samplers can see it. No config change, applies to all streams at once, and it mirrors what logs and metrics already do. PR to follow.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Manager.forStream and the span-name construction in internal/component/input/async_reader.go, processor/auto_observed.go, and output/async_writer.go. Trace how the provider reaches each stream, then verify that spans receive the stream identity at start and that the attribute is available for sampling and TraceQL or Jaeger filtering without a config change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.