github / github/gh-aw

[trajectory-grader] Implement end-to-end-lineage-completeness

Open
#60,592 0 comments 0 reactions 0 assignees View on GitHub
agentic-workflows automation graders observability trajectory-graders
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

> [!CAUTION]
> Protected files were modified in this change.
> This pull request is in `request-review` mode and requires explicit human scrutiny before merge.
>
> Protected files: `README.md`, `.github/workflows/shared/graders/README.md`, `.github/workflows/shared/graders/end-to-end-lineage-completeness.md`

Implements **`end-to-end-lineage-completeness`** — rank 13, Tier 2 ("needs explicit constraints/states/provenance/objectives") from the [trajectory grader catalog](../blob/main/.github/workflows/shared/graders/README.md).

## What it measures
For every final-output event (`kind: "safe_output"`) in the canonical Trajectory IR, performs a backward BFS over `provenanceEdges` to determine whether it is traceable — possibly through multiple intermediate actions — all the way back to an evidence/tool root (an `observations[]` or `toolCalls[]` id with no further incoming edge). Reports the fraction of final outputs with a complete end-to-end lineage. Higher is better.

## Why it's distinct from existing graders
- **`action-provenance-coverage`** (#14, not yet implemented) only checks whether a consequential action has a single recorded antecedent edge — a shallow, one-hop check.
- **`tool-output-consumption-rate`** (#12, implemented) measures forward consumption: whether observations were referenced by later actions.
- `end-to-end-lineage-completeness` is the only one of the three that walks the *full* provenance chain, so it distinguishes fully-grounded final outputs from outputs whose only recorded antecedent is itself a dead end (an intermediate action with no root behind it).
- None of the existing built-in graders (`tool-success-rate`, `retries`, `loops`, `trajectory-efficiency`, etc.) measure provenance/lineage at all.

## Required IR fields
- `events[]` — `kind === "safe_output"` and `.ref` to find final outputs
- `observations[].id`, `toolCalls[].id` — evidence/tool roots
- `provenanceEdges[].from` / `.to` — backward traversal edges
- `actions[].id` / `.type` — used only to label untraceable ids in `details`

Reports `not-applicable` (`passed: null`) rather than fabricating a value when there are no `safe_output` events, no observation/toolCall roots, or unresolvable ref ids.

## Catalog update
Flips `end-to-end-lineage-completeness` from `Not started` to `Implemented` in `shared/graders/README.md`. **14 of 25** catalog graders now implemented — see [the full catalog](../blob/main/.github/workflows/shared/graders/README.md) for the remaining list.

---

> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Original error:** The process '/usr/bin/git' failed with exit code 1
>
> **Workflow Run:** [View run details and download bundle artifact](https://github.com/github/gh-aw/actions/runs/34744101989)
>
> The bundle file is available in the `agent` artifact in the workflow run linked above.

Create the pull request manually

```sh
# Download the artifact from the workflow run
gh run download '34744101989' -n agent -D '/tmp/agent-34744101989'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-34744101989/aw-add-end-to-end-lineage-completeness-grader.bundle'
temp_ref='refs/bundles/create-pr-add-end-to-end-lineage-completeness-grader-08bc2022d0131815-58e248e6'
target_ref='refs/heads/add-end-to-end-lineage-completeness-grader-08bc2022d0131815'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'add-end-to-end-lineage-completeness-grader-08bc2022d0131815'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'add-end-to-end-lineage-completeness-grader-08bc2022d0131815'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement end-to-end-lineage-completeness' --base 'main' --head 'add-end-to-end-lineage-completeness-grader-08bc2022d0131815' --repo 'github/gh-aw'
```

> Generated by [🧮 Daily Trajectory Grader Implementer](https://github.com/github/gh-aw/actions/runs/34744101989) · copilot · auto · 157.5 AIC · ⌖ 10.4 AIC · ⊞ 8.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+%22gh-aw-workflow-id%3A+daily-trajectory-grader-implementer%22&type=pullrequests)
> - [x] expires on Sep 26, 2026, 11:10 PM UTC-08:00

Contributor guide

Open the contributing guide

Research direction

Start by reviewing .github/workflows/shared/graders/README.md and .github/workflows/shared/graders/end-to-end-lineage-completeness.md, then inspect the existing trajectory grader entry points and tests. Done means the grader is implemented for the stated IR fields and edge cases, and the catalog marks it Implemented; the issue also includes a workflow bundle and manual PR commands to inspect.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.