rivet-dev / rivet-dev/actors

fix(gasoline): bound traces from long-lived workflow loops

Open
#5,671 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
6.1k
Forks
250
Avg merge
2d 8h
Merged PRs (30d)
96

Description

Summary

Gasoline workflow loops can keep one OpenTelemetry trace ID for the full lifetime of a long-running workflow. Each loop iteration is instrumented as a child of the long-lived LoopBuilder::run / workflow span, so traces grow without a bound.

The current implementation is here:

https://github.com/rivet-dev/actors/blob/caed5835f/engine/packages/gasoline/src/builder/workflow/lupe.rs#L284

Evidence

Observed with Rivet Engine 2.3.0-rc.5 in self-hosted staging and production:

  • One block-only Tempo response contained 12,205 Rivet spans over 65 minutes and required 50,389,617 inspected bytes.
  • The visible iteration counters were already 7,650–7,778, showing that the trace belonged to a much older workflow parent.
  • The largest operation groups were txn (2,046), txn_attempt (2,045), and four signal-listening operations (1,269 each).
  • Over 36 hours, Tempo emitted 247 oversized-trace warnings from 3 trace IDs in staging and 151 warnings from 39 trace IDs in production.
  • The most repeated production trace had the same shape: 12,313 spans dominated by transaction and signal-loop operations.
  • These traces repeatedly hit Tempo's trace-size guard during compaction and contributed to ingester/compactor memory pressure.

The current public main still has the same inherited iteration span, and I could not find an existing issue or PR for this behavior.

Suggested fix

Start each workflow iteration as a new root trace and preserve causality by linking it to the workflow span:

let parent_span_ctx = tracing::Span::current()
    .context()
    .span()
    .span_context()
    .clone();
let iteration_span =
    tracing::info_span!(parent: None, "iteration", iteration=%previous_iteration);
iteration_span.add_link(parent_span_ctx);

let res = async {
    // existing iteration body
}
.instrument(iteration_span)
.await?;

This bounds each trace to one workflow iteration while retaining a causal link to the long-lived workflow. I have this change locally and rustfmt --check plus cargo check -p gasoline pass.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in engine/packages/gasoline/src/builder/workflow/lupe.rs around line 284, where workflow iterations inherit the long-lived span. Read the surrounding loop instrumentation, then run rustfmt --check and cargo check -p gasoline. Done means each iteration is bounded to its own trace while retaining a causal link to the workflow span.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.