githubnext / githubnext/agentics

repo-assist: Task 8 (Monthly Activity Summary) does not run deterministically despite being marked mandatory

Open
#372 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Makefile
Stars
952
Forks
139
Avg merge
6d 10h
Merged PRs (30d)
2

Description

## Summary

In `repo-assist.md`, Task 8 (Update Monthly Activity Summary Issue) is documented as mandatory every run:

> "Always do Task 8 (Update Monthly Activity Summary Issue) every run."

In practice, on the `tenstorrent/tt-metal` deployment, it is **not** running deterministically. Over the last 7 daily scheduled runs (2026-07-24 through 2026-07-30, all completed successfully), the monthly summary issue ([#50956](https://github.com/tenstorrent/tt-metal/issues/50956)) was only actually updated on 2 of them.

## Evidence

| Date | Run | Task 8 landed? |
|---|---|---|
| 07-24 | [30072973467](https://github.com/tenstorrent/tt-metal/actions/runs/30072973467) | ❌ Called `update_issue`; safe-outputs manifest logged it as processed, but no edit ever appears on the issue's timeline. Silently dropped. |
| 07-25 | [30147764461](https://github.com/tenstorrent/tt-metal/actions/runs/30147764461) | ✅ `add_comment` landed |
| 07-26 | [30191579550](https://github.com/tenstorrent/tt-metal/actions/runs/30191579550) | ❌ Trace shows it read the issue and planned "Task 8: Update monthly activity summary," then never called it |
| 07-27 | [30244126524](https://github.com/tenstorrent/tt-metal/actions/runs/30244126524) | ✅ `add_comment` landed |
| 07-28 | [30335618590](https://github.com/tenstorrent/tt-metal/actions/runs/30335618590) | ❌ `state.json` memory records `"task8_monthly_summary"` as done; actual `add_comment` calls that run targeted other issues/PRs (51294, 51296, 51024) instead |
| 07-29 | [30429064534](https://github.com/tenstorrent/tt-metal/actions/runs/30429064534) | ❌ Same pattern — reads #50956, reasons through "Task 8," then spends its comment budget on stale-PR nudges |
| 07-30 | [30520369864](https://github.com/tenstorrent/tt-metal/actions/runs/30520369864) | ❌ Same again |

Issue #50956's `updated_at` has been stuck at `2026-07-27T07:02:56Z` since — three consecutive daily runs produced zero visible output on it, despite each run's own reasoning trace explicitly citing "Task 8" as a planned step.

## Root cause (best guess)

"Always do Task 8" is a prose instruction competing for attention/turn budget against the round-robin task selection in the same prompt, not a structurally-guaranteed step in the compiled workflow. Nothing fails or flags the run if the write doesn't happen. The agent's own repo-memory (`state.json`) isn't validated against ground truth either — it records `task8_monthly_summary: done` even on runs where no comment/edit actually reached the issue. On 07-24, the `update_issue` safe-output specifically appears to fail silently (logged as processed, but no visible effect and no error).

## Question

Is there a supported pattern for making a task like this deterministic — e.g.:
- Splitting it into its own workflow/job that always runs, decoupled from the round-robin agent turn
- A safe-outputs guarantee/verification step that fails the run (or retries) if a declared "always do" output type wasn't actually emitted
- Reconciling agent-reported `state.json` progress against actual GitHub state rather than trusting the agent's self-report

Happy to help test a fix if there's a preferred approach.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with repo-assist.md Task 8, then compare the seven linked workflow runs, their traces, safe-outputs manifests, and state.json against issue #50956's timeline. Determine where the mandatory update can be skipped or silently dropped; done means Task 8 runs deterministically and failed or missing output is detected or retried.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, github-actions
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.