quickwit-oss / quickwit-oss/quickwit

Jaeger: Single lookback_period_hours causes performance vs functionality trade-off

Open
#5,887 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
The current Jaeger configuration in Quickwit uses a single lookback_period_hours parameter for all time-unbounded queries (get_services, get_operations, and get_trace operations). This creates a problem when you need different time windows for service/operation discovery versus trace retrieval operations.
The issue is that I cannot increase the current parameter because it would make the search page display too slow (service and operation discovery would take too long), but I need the find_trace functionality to work with a longer lookback period because it's very convenient for sharing traces between colleagues via trace links.

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  • Configure Quickwit with Jaeger endpoint enabled and a short lookback_period_hours (e.g., 24 hours) for acceptable UI performance
  • Use Jaeger UI to browse services and operations (this works well with short lookback period)
  • Try to share a trace link with a colleague for a trace that is older than the configured lookback period
  • The trace retrieval will fail or return incomplete results because it's using the same short lookback period
  • If you increase lookback_period_hours to fix trace sharing, the search page becomes unacceptably slow

Expected behavior
There should be separate configuration parameters:

  • One for service/operation discovery operations (get_services, get_operations) that can be set to a shorter period (e.g., 24-72 hours) for good UI performance
  • Another for trace retrieval operations (get_trace, find_traces) that can be set to a longer period (e.g., 30 days) to ensure trace links work properly for sharing traces between colleagues and accessing historical traces

Configuration:
jaeger: enable_endpoint: true lookback_period_hours: 24 # Can't increase this without making search page too slow max_trace_duration_secs: 3600 max_fetch_spans: 10000

Use case impact

  • Performance constraint: Cannot increase lookback_period_hours because service/operation discovery becomes too slow
  • Collaboration need: Need longer lookback for find_trace to share trace links between colleagues
  • Historical analysis: Need to access older traces for debugging and investigation purposes

Proposed solution
Split the lookback_period_hours into two separate parameters:

  • service_discovery_lookback_period_hours: For get_services and get_operations (default: 24-72 hours for performance)
  • trace_lookback_period_hours: For get_trace and trace retrieval operations (default: longer period, e.g., 30 days for trace sharing)

Maintain backward compatibility by keeping the existing lookback_period_hours as a fallback when the new parameters are not specified.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Jaeger endpoint configuration and the implementations of get_services, get_operations, get_trace, and find_traces. Trace how lookback_period_hours is applied, then define separate discovery and trace-retrieval periods with the existing setting as a fallback; done means short discovery queries and longer trace links work without breaking existing configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.