open-telemetry / open-telemetry/shared-workflows
pull-request-dashboard: replay corpus cannot validate routes end to end
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 7
- Forks
- 18
- Avg merge
- 22h 1m
- Merged PRs (30d)
- 75
Description
The 441 pull request replay corpus used to validate dashboard classification changes was collected without commits, so it cannot be used to compare whole-PR routes before and after a change.
normalize_events reads raw["commits"], and the harnesses work around this by passing {"commits": [], "review_comments": []}. That is fine for deriving discussion items and review threads, but it means compute_facts never sees pushes or CI, so route_pr cannot be evaluated. Any classification change can therefore only be measured at the discussion layer.
Concretely, the review-thread rewrite could be measured as "134 of 144 pull requests unchanged in whether any thread demands the author", but not as "N of 441 routes unchanged", which is the number that actually matters.
To close this:
- re-collect the corpus with
commitsand whatever elsecompute_factsreads (status checks, mergeable state) - add a replay that computes
route_prper pull request and diffs against therouterecorded in theotelbot/pull-request-dashboard-state/<repo>branches - keep the collector in the repo rather than as local scratch, so the corpus can be rebuilt
Until then, route-level claims about classification changes are not supported by evidence.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing normalize_events, compute_facts, and route_pr, then inspect the existing harnesses and route values in otelbot/pull-request-dashboard-state/ branches. Rebuild the 441-PR corpus with commits, status checks, and mergeable state, keep the collector in the repository, and add a replay that diffs computed routes against recorded routes. Done means route-level classification changes can be compared across the corpus.
Written by the indexing model from the issue text.
Assessment
- Domain
- devtools, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100