aspect-build / aspect-build/rules_js

[FR]: Full support for the pnpm file protocol including the ability to link packages outside the main Bazel repository

Open
#2,306 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Starlark
Stars
378
Forks
183
Avg merge
1d 9h
Merged PRs (30d)
32

Description

### What is the current behavior?

To my knowledge, `npm_translate_lock` currently has limited support for the `file:` protocol and requires that paths be within the main repository and thus immediately visible to Bazel.

### Describe the feature

I'd like to propose extending support to allow external local packages to be linked in the pnpm lockfile. This way devs can replace registry fetched npm dependencies with local versions on disk which is very useful for testing patches prior to publishing new versions when relying on 2p/3p dependencies which are not vendored inside their monorepo.

This could be achieved under the hood by backing local npm package references with a [`new_local_repository`](https://bazel.build/rules/lib/repo/local)rule and some default BUILD file content that exposes a `pkg` target. That `pkg` target can be an `npm_package` rule which by default would include all sources discoverable via the `npm pack` command.

In order for developers to make use of this feature they would simply replace the version of their 2p/3p dependency with a path to the local repository and then update their pnpm lockfile.

### Internal monorepo package JSON
```json
{
"name": "my-monorepo-app",
"dependencies": {
"lodash": "file:/Users/johndoe/repos/lodash"
}
...
}
```

### pnpm-lock.yaml
```yaml
importers:
packages/my-monorepo-app:
devDependencies:
'lodash':
specifier: file:/Users/johndoe/repos/lodash
version: file:/Users/johndoe/repos/lodash
```

If the `npm_translate_lock` repository rule detects a local path outside the main repository, it would first check to see if a Bazel repository already exists at that path in which case it would back the npm dependency with a `local_repository` rule and assume a `pkg` target has been defined. If no such repository exists, it would back the local link with a `new_local_repository` repo rule and run `npm pack --dry-run` to determine which files would be included when the package is published in order to mimic how the local repository would behave if it were being fetched from a registry.

This workflow is purely for local development purposes.

## Is this already possible with `replace_package` in `npm_translate_lock`?

Yes and no, by extending support for the file protocol to pick up local npm packages outside the main Bazel repo we allow local packages to participate in dependency resolution. `replace_package` overwrites packages post resolution and thus if developers are testing changes in 2p/3p that includes dependency version changes or the introduction of new dependencies they are not currently able to do so without manually updating their pnpm lockfile by some other means.

Currently, in my monorepo we developed a utility script which adds `replace_package` attributes to the `npm_translate_lock` tag class for generated repositories using `new_local_repository`. These attributes are generated from a custom configuration JSON file. The actual `package.json` files that declare these dependencies remain untouched. In order to take care of the limitation where these packages are not able to participate in depdendency resolution we overrwrite package.json contents for locally linked dependencies [using a readPackage hook](https://pnpm.io/pnpmfile) in our `.pnpmfile.cjs`. This is a bit hacky and it has the downside that developers might commit lockfiles that describe dependencies which are inconsistent with the registry state of truth.

Contributor guide

Open the contributing guide

Research direction

Start at npm_translate_lock's handling of the pnpm file: protocol and compare it with replace_package behavior. Review the proposed new_local_repository and local_repository paths, including npm pack --dry-run and the generated pkg target. Done means external local packages can participate in lockfile dependency resolution without the workaround described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.