hyperdxio / hyperdxio/hyperdx

[Feature Request] Trace-level AND search across spans

Open
#3,004 2 comments 0 reactions 1 assignee View on GitHub

@brandon-pereira is already working on this.

Since Sep 18, 2026.

external feature
Dominant language
TypeScript
Stars
9.9k
Forks
471
Avg merge
2d 4h
Merged PRs (30d)
117

Description

Summary

The search bar evaluates AND at the span/row level. There is no way to express a trace-level filter: return traces where property A appears on some span and property B appears on a (possibly different) span in the same trace.

That is a common query when correlation IDs are stamped once at the edge and service name (or other attributes) live on downstream spans.

Problem

A query like:

SpanAttributes.tuid:"abc123" AND ServiceName:checkout-service

Expected (Jaeger / Haystack / many APM UIs): traces that contain both predicates, even if they sit on different spans.

Actual: 0 results unless a single span has both fields.

This is easy to misread as a bug. The search bar is doing what it was built to do — filtering rows. The gap is that trace-level, cross-span AND is not expressible there.

Why this happens

Typical OpenTelemetry instrumentation stamps a correlation ID (request ID, tenant ID, user ID, etc.) on the gateway/entry span. Downstream services appear as separate spans in the same trace and often do not copy that ID onto every child.

So a real trace can look like:

Span ServiceName tuid
entry gateway-service abc123
child checkout-service (unset)
  • Trace-level AND matches this trace
  • Span-level AND matches nothing

Current workaround

Intersect two TraceId sets outside the where bar:

  1. TraceIds where any span has tuid = X
  2. TraceIds where any span has ServiceName = Y
  3. Fetch full traces for the intersection

That works against ClickHouse, but it is not something a user can do from the search bar.

Proposed behavior

Support a trace-scoped search mode (or syntax) where boolean operators apply at the trace:

  • A AND B → traces that contain a span matching A and a span matching B
  • Keep today’s span-level AND as an explicit mode, since it is still useful (“this span is slow and this span is checkout-service”)

Jaeger, Haystack, and several commercial APM products default to the trace-level interpretation for this class of query.

Additional context

Verified directly in ClickHouse on otel_traces-style data: the traces exist; no single row satisfies both predicates. Happy to share a minimal SQL sketch of the TraceId intersection if useful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.