[trajectory-grader] Implement end-to-end-lineage-completeness
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 541
- Avg merge
- 5h 48m
- Merged PRs (30d)
- 773
Description
> [!CAUTION]
> Protected files were modified in this change.
> This pull request is in `request-review` mode and requires explicit human scrutiny before merge.
>
> Protected files: `README.md`, `.github/workflows/shared/graders/README.md`, `.github/workflows/shared/graders/end-to-end-lineage-completeness.md`
Implements **`end-to-end-lineage-completeness`** — rank 13, Tier 2 ("needs explicit constraints/states/provenance/objectives") from the [trajectory grader catalog](../blob/main/.github/workflows/shared/graders/README.md).
## What it measures
For every final-output event (`kind: "safe_output"`) in the canonical Trajectory IR, performs a backward BFS over `provenanceEdges` to determine whether it is traceable — possibly through multiple intermediate actions — all the way back to an evidence/tool root (an `observations[]` or `toolCalls[]` id with no further incoming edge). Reports the fraction of final outputs with a complete end-to-end lineage. Higher is better.
## Why it's distinct from existing graders
- **`action-provenance-coverage`** (#14, not yet implemented) only checks whether a consequential action has a single recorded antecedent edge — a shallow, one-hop check.
- **`tool-output-consumption-rate`** (#12, implemented) measures forward consumption: whether observations were referenced by later actions.
- `end-to-end-lineage-completeness` is the only one of the three that walks the *full* provenance chain, so it distinguishes fully-grounded final outputs from outputs whose only recorded antecedent is itself a dead end (an intermediate action with no root behind it).
- None of the existing built-in graders (`tool-success-rate`, `retries`, `loops`, `trajectory-efficiency`, etc.) measure provenance/lineage at all.
## Required IR fields
- `events[]` — `kind === "safe_output"` and `.ref` to find final outputs
- `observations[].id`, `toolCalls[].id` — evidence/tool roots
- `provenanceEdges[].from` / `.to` — backward traversal edges
- `actions[].id` / `.type` — used only to label untraceable ids in `details`
Reports `not-applicable` (`passed: null`) rather than fabricating a value when there are no `safe_output` events, no observation/toolCall roots, or unresolvable ref ids.
## Catalog update
Flips `end-to-end-lineage-completeness` from `Not started` to `Implemented` in `shared/graders/README.md`. **14 of 25** catalog graders now implemented — see [the full catalog](../blob/main/.github/workflows/shared/graders/README.md) for the remaining list.
---
> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Original error:** The process '/usr/bin/git' failed with exit code 1
>
> **Workflow Run:** [View run details and download bundle artifact](https://github.com/github/gh-aw/actions/runs/34744101989)
>
> The bundle file is available in the `agent` artifact in the workflow run linked above.
Create the pull request manually
```sh
# Download the artifact from the workflow run
gh run download '34744101989' -n agent -D '/tmp/agent-34744101989'
# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-34744101989/aw-add-end-to-end-lineage-completeness-grader.bundle'
temp_ref='refs/bundles/create-pr-add-end-to-end-lineage-completeness-grader-08bc2022d0131815-58e248e6'
target_ref='refs/heads/add-end-to-end-lineage-completeness-grader-08bc2022d0131815'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'add-end-to-end-lineage-completeness-grader-08bc2022d0131815'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"
# Push the branch to the target remote
git push 'origin' 'add-end-to-end-lineage-completeness-grader-08bc2022d0131815'
# Create the pull request
gh pr create --title '[trajectory-grader] Implement end-to-end-lineage-completeness' --base 'main' --head 'add-end-to-end-lineage-completeness-grader-08bc2022d0131815' --repo 'github/gh-aw'
```
> Generated by [🧮 Daily Trajectory Grader Implementer](https://github.com/github/gh-aw/actions/runs/34744101989) · copilot · auto · 157.5 AIC · ⌖ 10.4 AIC · ⊞ 8.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+%22gh-aw-workflow-id%3A+daily-trajectory-grader-implementer%22&type=pullrequests)
> - [x] expires on Sep 26, 2026, 11:10 PM UTC-08:00
Contributor guide
Research direction
Start by reviewing .github/workflows/shared/graders/README.md and .github/workflows/shared/graders/end-to-end-lineage-completeness.md, then inspect the existing trajectory grader entry points and tests. Done means the grader is implemented for the stated IR fields and edge cases, and the catalog marks it Implemented; the issue also includes a workflow bundle and manual PR commands to inspect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100