aws / aws/deep-learning-containers

agent-fix.py: untruncated CI job logs sent to Bedrock; MAX_LOG_LINES only applied in dead code

Open
#6,518 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
559
Avg merge
1d 7h
Merged PRs (30d)
102

Description

## Summary

`scripts/ci/autocurrency/agent-fix.py` embeds **entire CI job logs, untruncated**, into the Bedrock prompt. The `MAX_LOG_LINES = 500` cap written for exactly this purpose is only applied inside a helper that nothing calls. The script also re-downloads the full workflow-run log archive once per failed job.

## Environment

- Repo version: `main` @ [`a09e9fd`](https://github.com/aws/deep-learning-containers/commit/a09e9fdc74956fdf0ae87f01c788fc8936fa6ea0)
- Runs on the CI host via `_prcheck.currency-fix.yml`
- Python 3.12, `MODEL_ID = us.anthropic.claude-opus-4-6-v1`, `MAX_TOKENS = 16384`

## Steps to reproduce

No AWS access needed — the defect is on the path between the GitHub API and the prompt:

```bash
git checkout a09e9fd
grep -n "MAX_LOG_LINES" scripts/ci/autocurrency/agent-fix.py
```

```
21:MAX_LOG_LINES = 500
182: if len(error_lines) > MAX_LOG_LINES:
185: return "\n".join(error_lines[:MAX_LOG_LINES]) or "No error patterns found in logs."
```

Both uses are inside `_extract_via_grep()` (defined L162). Now confirm nothing calls it:

```bash
grep -rn "_extract_via_grep\|detect_failed_jobs" .
```

Only the two definitions come back. The live path, `extract_failure_info()`, has no cap.

## 1. The log cap is never applied on the live path

[`agent-fix.py#L148-L150`](https://github.com/aws/deep-learning-containers/blob/a09e9fdc74956fdf0ae87f01c788fc8936fa6ea0/scripts/ci/autocurrency/agent-fix.py#L148-L150):

```python
log_lines = z.read(name).decode(errors="replace").splitlines()
results.append(f" Log ({name}, {len(log_lines)} lines):")
results.extend(f" {line}" for line in log_lines)
```

That string flows straight into `build_prompt()` → `call_bedrock()`. A GPU build or a vLLM upstream test job produces tens of thousands of log lines; a single `docker buildx` job log is routinely multiple MB. Consequences, in the order they surface:

- **The request is rejected.** Bedrock raises `ValidationException` when the input exceeds the model's context window. The exception is unhandled, so `agent-fix.py` dies with a traceback and the currency-fix workflow reports a failure that has nothing to do with the CI failure it was invoked to diagnose.
- **Cost.** Every retry re-sends the same payload — `MAX_LLM_RETRIES = 3` attempts, each carrying the full log.
- **Diagnosis quality.** The failure is almost always in the last few hundred lines. Burying it under megabytes of build chatter is what the 500-line cap was meant to prevent.

## 2. The log archive is downloaded once per failed job

The zip fetch sits **inside** the `for job in data.get("jobs", [])` loop, but the URL is per-**run** ([L134](https://github.com/aws/deep-learning-containers/blob/a09e9fdc74956fdf0ae87f01c788fc8936fa6ea0/scripts/ci/autocurrency/agent-fix.py#L134)):

```python
zip_url = f"https://api.github.com/repos/{repo}/actions/runs/{run_id}/logs"
```

With N failed jobs in one run, the same archive is downloaded, held in memory via `io.BytesIO`, and re-parsed N times. A currency PR failing `build-image`, `sanity-test` and `security-test` together pulls the identical archive three times.

## 3. Dead code

Neither `_extract_via_grep()` (L162) nor `detect_failed_jobs()` (L195) is called anywhere. Both take a `logs_dir` argument that no longer exists — `parse_args()` has no such flag and `_prcheck.currency-fix.yml` never passes one.

`main()` still carries a comment describing a fallback that was removed ([L417-L419](https://github.com/aws/deep-learning-containers/blob/a09e9fdc74956fdf0ae87f01c788fc8936fa6ea0/scripts/ci/autocurrency/agent-fix.py#L417-L419)):

```python
error_lines, api_failed_jobs = extract_failure_info(args.run_ids, args.token, args.repo)
# Use API-detected jobs if available, otherwise fall back to log filename detection
failed_jobs = api_failed_jobs
```

There is no fallback. This appears to be how the cap went missing: the truncation lived in the path that was replaced, and the replacement never picked it up.

## Suggested fix

- Cap what reaches the prompt at `MAX_LOG_LINES`, keeping the **tail** of each job log with an explicit `... N earlier lines omitted ...` marker so the model knows the log was clipped.
- Fetch each run's log archive at most once.
- Remove the dead helpers and the stale comment.

**Correction (edited):** this issue originally said no PR had been opened. I have since opened #6521 with the patch, so that line was no longer true and has been replaced. I am aware CONTRIBUTING.md asks external contributors not to open PRs — please close #6521 without review if that is the standing policy and treat this issue as the report. Patch also on my fork: https://github.com/Adityaj0/deep-learning-containers/pull/6

Contributor guide

Open the contributing guide

Research direction

Start in scripts/ci/autocurrency/agent-fix.py at extract_failure_info(), build_prompt(), and the unused _extract_via_grep() and detect_failed_jobs() helpers; also read _prcheck.currency-fix.yml. Done means prompt logs are capped to the specified tail, each run archive is fetched once, and the obsolete helpers and stale fallback comment are removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, python
Domain
ai-infra-agents, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.