NVIDIA / NVIDIA/NemoClaw

ci: 15-minute CLI shard ceiling cancels otherwise healthy PR suites

Open
#9,982 5 comments 0 reactions 0 assignees View on GitHub
area: ci area: e2e ci-failure
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

The PR CLI coverage matrix has outgrown or intermittently exceeds its 15-minute per-job ceiling. Multiple unrelated PRs on the same base have several shards terminated simultaneously at the limit, producing no Vitest blob and forcing the aggregate `cli-tests` and `checks` jobs red without a test assertion failure.

## Evidence

### PR #9932

- Base: `7601ab27a82dfdb67c933c9e24612e52f8c367e4`
- Run: 32606449660
- Canceled at approximately 15 minutes: shards 1, 2, 5, 7, and 12
- Seven other shards passed.
- Each canceled log ends with `The operation was canceled.` during Vitest, followed by `No files were found` for the report artifact.

### Unrelated PR #9950

- Same base: `7601ab27a82dfdb67c933c9e24612e52f8c367e4`
- Run: 32606363353
- Started about two minutes earlier.
- Canceled at the same approximately 15-minute ceiling: shards 2, 4, 5, 6, 7, and 8
- Six other shards passed.

The different canceled-shard sets on unrelated changes indicate a CI duration/variance problem rather than a deterministic test assertion regression.

## Current configuration

Both `.github/workflows/pr.yaml` and `.github/workflows/main.yaml` set `timeout-minutes: 15` for `cli-test-shards`. The composite action installs dependencies, builds the plugin and CLI, then runs coverage before that shared job deadline.

## Expected behavior

The authoritative broad gate should finish all 12 shard reports with sufficient headroom. Options include increasing the job timeout, moving setup/build outside the measured shard budget, refreshing timing hints/shard allocation, or splitting the heaviest files.

## Validation request

Measure p95 durations for each shard on the current main test roster and set a bounded timeout with operational headroom. Preserve the aggregate fail-closed behavior for genuine test failures.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/pr.yaml and .github/workflows/main.yaml, then inspect the cli-test-shards composite action and recent shard timings. Measure p95 duration for each of the 12 shards on the current main test roster. Done means the authoritative gate completes all shard reports with bounded timeout headroom while still failing for genuine test failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.