microsoft / microsoft/lage

Unable to use custom cacheStorageConfig Provider

Open
#728 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
816
Forks
88
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
When trying to add my own custom cacheStorageConfig Provider (according to backfill docs) run job will fail with

DOMException [DataCloneError]: provider(logger, cwd) { 
    return new CustomStorageProvider(logger, cwd)
} could not be cloned.
at new DOMException (node:internal/per_context/domexception:53:5)
at new Worker (node:internal/worker:266:17)
at S8.TL (.../node_modules/lage/dist/lage.js:300:4009)
at new S8 (.../node_modules/lage/dist/lage.js:300:3932)
at C8.addNewWorker (.../node_modules/lage/dist/lage.js:300:6688)
at C8.createInitialWorkers (.../node_modules/lage/dist/lage.js:300:6572)
at new C8 (.../node_modules/lage/dist/lage.js:300:7970)
at new R8 (.../node_modules/lage/dist/lage.js:300:9635)
at new lE (.../node_modules/lage/dist/lage.js:345:7127)
at UAe (.../node_modules/lage/dist/lage.js:350:9245)

It looks like the task-runner is trying to serialize the function and hand it over to worker

To Reproduce
Steps to reproduce the behavior:

  1. Have a minimal monorepo setup with a build script
  2. Create a minimal lage.config.js with custom storage provider
class CustomStorageProvider {
    async fetch(hash) {
        console.log(hash)

        return false
    }

    async put(hash, filesToCache) {
        console.log(hash, filesToCache)
    }
}

module.exports = {
    cacheOptions: {
        cacheStorageConfig: {
            provider(logger, cwd) {
                return new CustomStorageProvider(logger, cwd)
            },
        },
        environmentGlob: ['.github/**'],
        skipLocalCache: false,
        writeRemoteCache: true,
    },
    ignore: ['*.md'],
    npmClient: 'yarn',
    pipeline: {
        build: {
            inputs: ['./src/**/*', '^./src/**/*']
        },
    },
}
  1. Add lage to build script "build": "lage build"
  2. Run build yarn run build

Expected behavior
I expected to be able to add a custom cache provider. My original goal was to use cache-github-action plugin but that one also doesn't work as expected (same clone error).

Environment/Troubleshooting

  System:
    OS: macOS 14.1.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 23.66 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 4.1.0 - ~/.nvm/versions/node/v20.10.0/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm
  Managers:
    CocoaPods: 1.11.2 - /usr/local/bin/pod
    Homebrew: 4.2.10 - /usr/local/bin/brew
    pip3: 22.3.1 - ~/.pyenv/shims/pip3
    RubyGems: 3.0.3.1 - /usr/bin/gem
  Utilities:
    Make: 3.81 - /usr/bin/make
    GCC: 15.0.0 - /usr/bin/gcc
    Git: 2.39.3 - /usr/bin/git
    Clang: 15.0.0 - /usr/bin/clang
    Curl: 8.1.2 - /usr/bin/curl
  Servers:
    Apache: 2.4.56 - /usr/sbin/apachectl
  Virtualization:
    Docker: 24.0.6 - /usr/local/bin/docker
  SDKs:
    iOS SDK:
      Platforms: DriverKit 23.2, iOS 17.2, macOS 14.2, tvOS 17.2, visionOS 1.0, watchOS 10.2
    Android SDK:
      API Levels: 28, 29, 30, 31, 33
      Build Tools: 28.0.3, 29.0.2, 30.0.2, 30.0.3, 33.0.0
      System Images: android-23 | Intel x86 Atom_64, android-25 | Google Play Intel x86 Atom, android-29 | Google APIs Intel x86 Atom, android-29 | Google Play Intel x86 Atom, android-30 | Google APIs Intel x86 Atom, android-30 | Google APIs Intel x86 Atom_64, android-31 | Google APIs Intel x86 Atom_64
  IDEs:
    Android Studio: 2021.1 AI-211.7628.21.2111.8193401
    Vim: 9.0 - /usr/bin/vim
    Xcode: 15.2/15C500b - /usr/bin/xcodebuild
  Languages:
    Bash: 3.2.57 - /bin/bash
    Go: 1.17.13 - /usr/local/go/bin/go
    Java: 11.0.21 - /usr/bin/javac
    Perl: 5.30.3 - /usr/bin/perl
    Protoc: 24.4 - /usr/local/bin/protoc
    Python: 3.11.3 - /Users/mid/.pyenv/shims/python
    Python3: 3.11.3 - /Users/mid/.pyenv/shims/python3
    Ruby: 2.6.10 - /usr/bin/ruby
  Databases:
    SQLite: 3.39.5 - /usr/bin/sqlite3
  Browsers:
    Brave Browser: 99.1.36.109
    Chrome: 119.0.6045.159
    Safari: 17.1
  Monorepos:
    Yarn Workspaces: 4.1.0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the minimal monorepo and lage.config.js shown in the issue, then inspect the worker setup around the stack trace in node_modules/lage/dist/lage.js. Check how cacheStorageConfig.provider and the cache-github-actions plugin are passed to workers; done means a custom provider can run without a DataCloneError.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.