aws / aws/jsii

Bundled dependencies in monorepo contain ignored files when using --pack-command

Open
#4,436 0 comments 0 reactions 0 assignees View on GitHub
bug p2
Dominant language
TypeScript
Stars
2.9k
Forks
267
Avg merge
1d 25m
Merged PRs (30d)
14

Description

### Describe the bug

Custom --pack-commands may not work correctly in monorepos with bundled dependencies. This is caused by the package manager losing important context when the copy is made for custom pack commands. This ultimately inflates the tarball by including files which should not be published.

In the project where I first encountered this a ~3MB tarball inflates to over 200MB as it starts including all source files, ignored files, test reports, and more.

This could be a security vulnerability for some users. Because this bug is caused by publishing files that are in .npmignore users may think they have their secrets ignored and unknowingly end up publishing them anyway.

### Expected Behavior

Setting the pack command to "pnpm pack" produces results similar to manually running "pnpm pack".

### Current Behavior

Using "pnpm pack" as the pack command causes bundled dependencies to contain the full source of the dependency rather than the output that should be packaged. I believe the reason for this is that since "pnpm pack" is now being run inside a copy of the package in a temp directory it is no longer able to identify that those packages belong to the same workspace and so need to recursively be packed, not just copied in.

I'm presently working around this issue by using `--pack-command="cd \"$PWD\" && pnpm pack --pack-destination \"$(mktemp -d)\""` to ignore the copy that was made and pack in the original location instead.

### Reproduction Steps

Run the following in an empty directory. Expects that npm, pnpm, and standard utils are available.
```
pnpm init
printf "packages:\n - 'packages/*'" > pnpm-workspace.yaml
mkdir -p packages/foo packages/bar
(cd packages/foo && pnpm init)
echo "/ignored" > packages/foo/.npmignore
touch packages/foo/ignored
(cd packages/bar && pnpm init)
(cd packages/bar && pnpm add foo --workspace && pnpm add -D jsii jsii-pacmak && npm pkg set 'bundledDependencies[]'=foo)
(cd packages/bar && npm pkg set --force 'author.name'=unused 'author.email'=unused 'repository.type'=git 'repository.url'=unused 'types'='lib/index.d.ts')
(cd packages/bar && npm pkg set --json 'jsii'='{"outdir":"dist","targets":{},"tsc":{"outDir":"lib","rootDir":"src"}}')
mkdir -p packages/bar/src
echo "export class Bar {}" > packages/bar/src/index.ts
(cd packages/bar && npx jsii && npx jsii-pacmak --pack-command="pnpm pack")
tar -tf 'packages/bar/dist/js/bar@1.0.0.jsii.tgz'
```
when complete observe that files that should not be present, such as "package/node_modules/foo/ignored", are present in the tarball. If you replace the text `pnpm pack` with `cd \"$PWD\" && pnpm pack --pack-destination \"$(mktemp -d)\"` then the issue will go away, although a wasteful copy will still occur.

The above was executed on both AL2 and an M1 Mac.

### Possible Solution

As can be observed by the custom command which `cd`s back to `$PWD` to avoid using the copy, not performing the copy present on https://github.com/aws/jsii/blob/main/packages/jsii-pacmak/lib/packaging.ts#L71-L72 can be used to solve the problem. Simply removing the copying though brings back the parallelism issue mentioned in the comment.

My preference as a consumer of JSII would be to not need to know I need to specify a custom pack-command for pnpm at all though. The most popular package managers could be detected via lock files and independently correctly handled. Sadly, this doesn't perfectly scale for every possible case as more package managers are created though.

### Additional Information/Context

_No response_

### SDK version used

1.93.0

### Environment details (OS name and version, etc.)

Amazon AL2

Contributor guide

Open the contributing guide

Research direction

Start in packages/jsii-pacmak/lib/packaging.ts, especially the copy operation referenced around lines 71-72, and reproduce the issue with the provided pnpm workspace commands and custom --pack-command. Inspect the generated packages/bar/dist/js/bar@1.0.0.jsii.tgz with tar -tf. Done means bundled dependencies contain only packable output and exclude ignored files such as package/node_modules/foo/ignored without reintroducing the parallelism problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.