pytorch / pytorch/pytorch

Investigate why XPU CI logs and artifacts are not being uploaded

Open
#193,741 3 comments 0 reactions 1 assignee Claimed by @chuanqi129 View on GitHub
bot-triaged module: ci module: xpu triaged
Dominant language
Python
Stars
103k
Forks
29.6k
PR merge metrics
PR metrics pending

Description

## Summary

We should investigate why logs are not being uploaded

## AI-generated audit evidence

> The following analysis was generated from a review of scheduled `pytorch/pytorch` XPU CI records and representative job logs. It is included as supporting evidence for the human concern above.
>
> ### Observed behavior
>
> XPU CI jobs can finish successfully even when their log or result upload steps fail. In a 14-day scheduled-main audit window ending August 16, 2026:
>
> - 352 successful `xpu` correctness jobs contained upload failures:
> - 293 S3 `PutObject` permission failures
> - 59 missing-`boto3` failures
> - 108 successful XPU Inductor-performance jobs contained upload failures:
> - 98 S3 permission failures
> - 5 missing-`boto3` failures
> - 5 expired-token failures
>
> The upload actions can report `outcome=failure` while the final step or job has `conclusion=success`. This allows CI to appear healthy even when logs or benchmark results are unavailable.
>
> Relevant areas include `.github/workflows/_xpu-test.yml`, `.github/workflows/inductor-perf-test-nightly-xpu.yml`, and the shared artifact-upload actions they invoke.
>
> ### Suggested investigation
>
> - Determine which artifacts should be routed to GitHub Actions and which require S3.
> - Use a hermetic Python invocation for host-side upload scripts, such as the existing pinned `uv run --no-project --with boto3==1.35.42` pattern.
> - Verify that S3 uploads assume a write-capable role.
> - Refresh credentials immediately before uploads following long-running tests.
> - Ensure that failure of every configured upload destination cannot remain silently green.
>
> ### Suggested completion criteria
>
> - No missing-`boto3`, S3 access-denied, or expired-token upload failures in three complete XPU cycles.
> - Expected logs and benchmark artifacts are present for each completed job.
> - An intentionally broken upload produces a visible degraded or failing result rather than an unqualified green conclusion.

cc @malfet @pytorch/pytorch-dev-infra @gujinghui @EikanWang @fengyuan14 @guangyey

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.