actions / actions/runner

Action runner ignores SIGPIPE and causes shell script with redirections to hang

Open
#2,684 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug keep
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
When running a script (https://github.com/btrfs/linux/actions/runs/5490218255/workflow) which runs fstests on a self-hosted VM, one of the tests will hang. This test is doing something like this

_fail() {
echo $1
exit 1
}

od /dev/urandom | dd of=/some/file bs=1M count=10 > somefile.txt 2>&1 || _fail "dd failed"

The boxes executing this line get stuck here, if you look at the /proc/pid/fd listing you can see

[root@xfstests6 fstests]# ll /proc/33527/fd
total 0
lr-x------ 1 root root 64 Jul 7 21:13 0 -> 'pipe:[51951]'
l-wx------ 1 root root 64 Jul 7 21:13 1 -> /root/fstests/results/btrfs_block_group_tree/btrfs/016.full
l-wx------ 1 root root 64 Jul 7 21:13 10 -> /tmp/30763.out
lr-x------ 1 root root 64 Jul 7 21:13 106 -> 'pipe:[53318]'
l-wx------ 1 root root 64 Jul 7 21:13 11 -> /tmp/30763.out
l-wx------ 1 root root 64 Jul 7 21:13 112 -> 'pipe:[53319]'
l-wx------ 1 root root 64 Jul 7 21:13 2 -> /root/fstests/results/btrfs_block_group_tree/btrfs/016.full
lr-x------ 1 root root 64 Jul 7 21:13 255 -> /root/fstests/tests/btrfs/016

and the ps auxw show

root 30742 5.7 2.6 3707952 102100 ? Sl 21:12 0:01 /root/actions-runner/bin/Runner.Worker spawnclient 106 112

so we're using those pipe's to capture the output of the job.

If I change this command to not do the redirects everything works fine.

Also I had a version of this runner that ssh'ed into the vm from a different machine to run this command, and this worked fine. It only breaks when the runner is on the machine that is trying to run this testsuite.

To Reproduce
Steps to reproduce the behavior:

  1. Start a local runner with fstests setup for btrfs (I can provide a series of steps to set this up if needed.)
  2. Create a workflow that runs ./check btrfs/016
  3. The task will hang

Expected behavior
The task should run just fine

Runner Version and Platform

actions-runner-linux-x64-2.305.0

OS of the machine running the runner? OSX/Windows/Linux/...
Fedora 38

What's not working?

The workflow is hanging when the runner is local to the machine running the testsuite.

Job Log Output

You can see my attempt here https://github.com/btrfs/linux/actions/runs/5490218255.

Runner and Worker's Diagnostic Logs

The logs don't show anything helpful, but I can provide them if needed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Runner.Worker spawnclient entry point and reproduce the Fedora 38 workflow using the shown od|dd command with output redirections. Inspect how the local runner captures these pipes and handles SIGPIPE. Done means the fstests workflow completes instead of hanging, while the redirected command still reports failures through the shown _fail path.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, shell
Domain
ci-cd, devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.