actions / actions/toolkit

Calling glob() on a directory of 200k files produces a RangeError: Maximum call stack size exceeded

Open
#1,545 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
5.9k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

Thank you 🙇‍♀ for wanting to create an issue in this repository. Before you do, please ensure you are filing the issue in the right place. Issues should only be opened on if the issue relates to code in this repository.

If your issue is relevant to this repository, please include the information below:

Describe the bug

Calling globber.glob on a directory with 200k files produces this error:

Run actions/github-script@v6.4.1
RangeError: Maximum call stack size exceeded
    at DefaultGlobber.globGenerator_1 (/home/runner/work/_actions/actions/github-script/v6.4.1/dist/index.js:3333:27)
    at globGenerator_1.next (<anonymous>)
    at resume (/home/runner/work/_actions/actions/github-script/v6.4.1/dist/index.js:3222:44)
Error: Unhandled error: RangeError: Maximum call stack size exceeded
    at fulfill (/home/runner/work/_actions/actions/github-script/v6.4.1/dist/index.js:3224:31)

To Reproduce

Here is a workflow that reproduces the issue:

name: Reproduce globbing issue w/ too many files
on:
  pull_request:
    branches:
      - main
jobs:
  reproduce-glob-issue-too-many-files:
    name: Reproduce globbing issue w/ too many files
    runs-on:
      - ubuntu-latest
    steps:
      - name: Generate 100k files
        run: |
          mkdir -p ${{ runner.temp }}/test
          for i in {1..100000}
          do
            echo $i > ${{ runner.temp }}/test/f$i.txt
          done
      # This one is fine.
      - name: Glob 100k files
        uses: actions/github-script@v6.4.1
        with:
          script: |
            const path = `${process.env.RUNNER_TEMP}/test/*.txt`
            const globber = await glob.create(path)
            const files = await globber.glob()
            console.log(files.length)
      - name: Generate additional 100k files
        run: |
          mkdir -p ${{ runner.temp }}/test
          for i in {100000..200000}
          do
            echo $i > ${{ runner.temp }}/test/f$i.txt
          done
      # This one fails with the RangeError.
      - name: Glob 200k files
        uses: actions/github-script@v6.4.1
        with:
          script: |
            const path = `${process.env.RUNNER_TEMP}/test/*.txt`
            const globber = await glob.create(path)
            const files = await globber.glob()
            console.log(files.length)

Here is a public example where I've run this workflow and it has failed:

Expected behavior

Calling .glob should not fail with a RangeError, or a known limit should be documented, or a workaround should be documented.

Screenshots

image

Desktop (please complete the following information):

Not relevant, see additional context for relevant details

Smartphone (please complete the following information):

Not relevant, see additional context for relevant details

Additional context

I've been able to reproduce this on public Github using github-script 6.4.1 and on Github Enterprise using github-script 6.3.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the globber.glob entry point and the DefaultGlobber.globGenerator stack frame shown in the report. Run the supplied GitHub Actions workflow to compare globbing 100k and 200k files. Done means the 200k-file case no longer raises RangeError, or the supported limit or workaround is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.