SciML / SciML/.github

Repeated self-hosted runner disconnects in SciMLOperators downstream LinearSolve cell

Open
#116 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
6
Forks
5
Avg merge
2d 1h
Merged PRs (30d)
8

Description

Summary

The LinearSolve.jl/InterfaceII cell in SciMLOperators' downstream workflow has repeatedly ended because its self-hosted runner lost communication with GitHub Actions. This is an infrastructure issue: the affected job has no Julia test failure to debug, and the same workflow cell later passed on unchanged workflow configuration.

Affected cell:

Downstream LinearSolve.jl/InterfaceII / Downstream Tests - InterfaceII

The workflow invokes SciML/.github/.github/workflows/downstream.yml@v1.

Affected runs

GitHub attached the same runner-level annotation to the LinearSolve cell in each run:

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

In run 29223216729, the downstream test step was still reported as in_progress when the job closed. The job log blob now returns 404, so there is no package-level exception or failed assertion available to investigate. That run started at 2026-07-13 04:11:26 UTC and ended at 04:39:28 UTC.

Control run and bisectability

The same cell passed in https://github.com/SciML/SciMLOperators.jl/actions/runs/29235992715, which started at 2026-07-13 08:35:48 UTC and completed at 08:59:24 UTC. Its log contains substantive Test Summary output. The SciMLOperators commits between the failed run and this green run did not change the downstream workflow or its LinearSolve dependency configuration.

There is therefore no source commit to bisect for these disconnects: GitHub reports termination outside the Julia test process, and the same reusable workflow/cell subsequently completed successfully without a relevant source or workflow fix.

Requested investigation

Please inspect the self-hosted runner service, lifecycle, network connectivity, and resource/OOM logs around the affected timestamps. In particular, it would be useful to determine why this LinearSolve downstream cell repeatedly loses its runner after roughly 20–30 minutes while sibling downstream cells complete.

This issue intentionally tracks only the runner disconnects. Any downstream group-selection cleanup should be handled separately.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the six linked failed jobs and the successful control run, focusing on Downstream LinearSolve.jl/InterfaceII / Downstream Tests - InterfaceII and the reusable workflow SciML/.github/.github/workflows/downstream.yml@v1. Inspect the self-hosted runner service, lifecycle, network, CPU/memory, and OOM logs around the listed timestamps; done means identifying the runner disconnect cause or documenting the relevant evidence and remediation.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, julia
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.