open-telemetry / open-telemetry/shared-workflows

pull-request-dashboard: replay corpus cannot validate routes end to end

Open
#192 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
7
Forks
18
Avg merge
22h 1m
Merged PRs (30d)
75

Description

The 441 pull request replay corpus used to validate dashboard classification changes was collected without commits, so it cannot be used to compare whole-PR routes before and after a change.

normalize_events reads raw["commits"], and the harnesses work around this by passing {"commits": [], "review_comments": []}. That is fine for deriving discussion items and review threads, but it means compute_facts never sees pushes or CI, so route_pr cannot be evaluated. Any classification change can therefore only be measured at the discussion layer.

Concretely, the review-thread rewrite could be measured as "134 of 144 pull requests unchanged in whether any thread demands the author", but not as "N of 441 routes unchanged", which is the number that actually matters.

To close this:

  • re-collect the corpus with commits and whatever else compute_facts reads (status checks, mergeable state)
  • add a replay that computes route_pr per pull request and diffs against the route recorded in the otelbot/pull-request-dashboard-state/<repo> branches
  • keep the collector in the repo rather than as local scratch, so the corpus can be rebuilt

Until then, route-level claims about classification changes are not supported by evidence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing normalize_events, compute_facts, and route_pr, then inspect the existing harnesses and route values in otelbot/pull-request-dashboard-state/ branches. Rebuild the 441-PR corpus with commits, status checks, and mergeable state, keep the collector in the repository, and add a replay that diffs computed routes against recorded routes. Done means route-level classification changes can be compared across the corpus.

Written by the indexing model from the issue text.

Assessment

Domain
devtools, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.