elastic / elastic/ai-github-actions
[autonomy-atomicity] PR-readiness test enforces lockstep internals across two safe-output fragments
- Dominant language
- Python
- Stars
- 11
- Forks
- 16
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 31
Description
## Autonomy / Atomicity Findings
### 1. `test_fragments_share_common_structure` couples two independent safe-input scripts via string markers
**Category:** Over-broad test
**File(s):**
- `tests/test_safe_input_ready_to_make_pr.py` (lines 123-130)
- `.github/workflows/gh-aw-fragments/safe-output-push-to-pr.md` (safe-input Python block)
- `.github/workflows/gh-aw-fragments/safe-output-create-pr.md` (safe-input Python block)
**Problem:**
`tests/test_safe_input_ready_to_make_pr.py` asserts that both extracted Python scripts contain the same implementation markers:
- `contributing = find(`
- `diff_line_count`
- `json.dumps`
- `self-review`
This creates a lockstep coupling between two different workflows (`ready-to-push-to-pr` vs `ready-to-make-pr`). If one script is correctly refactored (renamed variables, restructured logic, equivalent behavior), the shared marker test fails until the other script is also edited to match internals. In concurrent work, unrelated changes in one path can break CI for another path and force rebases/coordination.
**Suggested fix:**
Replace marker-string coupling with behavior-oriented assertions per fragment:
- Validate each fragment’s output schema and required checklist semantics independently.
- Keep only truly shared behavior checks in shared helpers.
- Remove assertions that require matching internal variable names or textual structure across both scripts.
**Duplicate check:**
- Closest open issue: `#1771` (`safe-input extraction helper relies on YAML key order across PR readiness tests`).
- This finding is distinct: it is about cross-fragment **string-level lockstep coupling** in `test_fragments_share_common_structure`, not YAML key-order dependence.
## Suggested Actions
- [ ] Refactor `test_fragments_share_common_structure` to assert outcomes/contract instead of implementation markers.
- [ ] Add per-fragment tests that allow internal refactors without requiring synchronized textual changes in both scripts.
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Trigger Autonomy Atomicity Analyzer](https://github.com/elastic/ai-github-actions/actions/runs/31818781875)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
Contributor guide
Research direction
Start with tests/test_safe_input_ready_to_make_pr.py, especially test_fragments_share_common_structure and lines 123-130, then inspect the safe-input Python blocks in the two named workflow fragments. Replace cross-fragment marker coupling with independent output-schema and checklist-semantic assertions, and add per-fragment coverage. Done means valid internal refactors in one fragment no longer require matching textual changes in the other.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100