fix(gasoline): bound traces from long-lived workflow loops
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.1k
- Forks
- 250
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 96
Description
Summary
Gasoline workflow loops can keep one OpenTelemetry trace ID for the full lifetime of a long-running workflow. Each loop iteration is instrumented as a child of the long-lived LoopBuilder::run / workflow span, so traces grow without a bound.
The current implementation is here:
Evidence
Observed with Rivet Engine 2.3.0-rc.5 in self-hosted staging and production:
- One block-only Tempo response contained 12,205 Rivet spans over 65 minutes and required 50,389,617 inspected bytes.
- The visible iteration counters were already 7,650–7,778, showing that the trace belonged to a much older workflow parent.
- The largest operation groups were
txn(2,046),txn_attempt(2,045), and four signal-listening operations (1,269 each). - Over 36 hours, Tempo emitted 247 oversized-trace warnings from 3 trace IDs in staging and 151 warnings from 39 trace IDs in production.
- The most repeated production trace had the same shape: 12,313 spans dominated by transaction and signal-loop operations.
- These traces repeatedly hit Tempo's trace-size guard during compaction and contributed to ingester/compactor memory pressure.
The current public main still has the same inherited iteration span, and I could not find an existing issue or PR for this behavior.
Suggested fix
Start each workflow iteration as a new root trace and preserve causality by linking it to the workflow span:
let parent_span_ctx = tracing::Span::current()
.context()
.span()
.span_context()
.clone();
let iteration_span =
tracing::info_span!(parent: None, "iteration", iteration=%previous_iteration);
iteration_span.add_link(parent_span_ctx);
let res = async {
// existing iteration body
}
.instrument(iteration_span)
.await?;
This bounds each trace to one workflow iteration while retaining a causal link to the long-lived workflow. I have this change locally and rustfmt --check plus cargo check -p gasoline pass.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in engine/packages/gasoline/src/builder/workflow/lupe.rs around line 284, where workflow iterations inherit the long-lived span. Read the surrounding loop instrumentation, then run rustfmt --check and cargo check -p gasoline. Done means each iteration is bounded to its own trace while retaining a causal link to the workflow span.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100