actions / actions/runner

Long log lines slow down the runner

Open
#1,031 15 comments 10 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug papercut Runner Bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
Outputting very long log lines seems to cause the runner to slow down significantly.
A step that takes less than a second to actually execute can easily take 10s of minutes or even hours to be considered complete, often hitting timeouts.

https://github.com/hamishforbes/actions-test
This repo has a super simple python script that outputs the current timestamp and "a" repeated for various lengths.

Turn on timestamps and you can see a discrepancy.
Once a log message gets up to about 6k characters it looks like the runner starts to take a few seconds to process.
At 100k characters this can be 10 minutes or so

Running the script locally on a macbook pro, even with 100k characters, completes within a couple hundred ms.

Runner timestamp              | Python timestamp
Tue, 30 Mar 2021 01:34:19 GMT | 2021-03-30 01:34:16.831361 - Message length: 10240 - aaaaa{snip}
Tue, 30 Mar 2021 01:34:30 GMT | 2021-03-30 01:34:16.831413 - Message length: 20480 - aaaaa{snip}

To Reproduce
Copy script and workflow from above repo :)

Expected behavior
I would expect the runner to be able to handle long log lines even if truncating them.
At the very least it shouldn't delay how long the step takes to run, the real world workflow that caused this was completing its test suite in < 10 mins but the step would never finish in actions because it would hit timeouts

Runner Version and Platform

Github hosted but tested with the latest self-hosted runner as well

Seems to affect both steps executed in docker and those directly on ubuntu-latest, although slightly less severe.
https://github.com/hamishforbes/actions-test/tree/no_docker

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction script and workflow in the linked actions-test repository, including the no_docker variant, and compare runner timestamps with the Python timestamps at increasing message lengths. Confirm the delay for long log lines and verify that handling or truncating them no longer makes the step exceed its actual execution time or hit a timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, python
Domain
ci-cd, devtools, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.